Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:55.765883 21441 test_util.cc:268] Using random seed: 1297674323
I20240503 14:11:55.777082 21441 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:11:55.777151 21441 ts_itest-base.cc:113] --------------
I20240503 14:11:55.777208 21441 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:11:55.777294 21441 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:11:55.777365 21441 ts_itest-base.cc:116] --------------
2024-05-03T14:11:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:11:55Z Disabled control of system clock
I20240503 14:11:55.800092 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43789
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:43789 with env {}
W20240503 14:11:56.125046 21453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.125320 21453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.125412 21453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.132565 21453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:56.132663 21453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.132702 21453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:56.132755 21453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:56.141566 21453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:43789
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.142854 21453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.144541 21453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.155624 21461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.156387 21459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.157225 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.157856 21453 server_base.cc:1034] running on GCE node
I20240503 14:11:56.158632 21453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.159897 21453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.161121 21453 hybrid_clock.cc:648] HybridClock initialized: now 1714745516161053 us; error 33 us; skew 500 ppm
I20240503 14:11:56.161495 21453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.164167 21453 webserver.cc:460] Webserver started at http://127.20.240.126:41035/ using document root <none> and password file <none>
I20240503 14:11:56.164779 21453 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.164892 21453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.165144 21453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.167670 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e21746e837d5438288989b67557befc1"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-xhmt"
I20240503 14:11:56.168336 21453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e21746e837d5438288989b67557befc1"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-xhmt"
I20240503 14:11:56.173007 21453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 14:11:56.176116 21467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.177281 21453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:11:56.177435 21453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e21746e837d5438288989b67557befc1"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-xhmt"
I20240503 14:11:56.178136 21453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.226084 21453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.226917 21453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.227207 21453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.245102 21453 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:43789
I20240503 14:11:56.245111 21518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:43789 every 8 connection(s)
I20240503 14:11:56.246482 21453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:11:56.249984 21519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:56.256466 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 21453
I20240503 14:11:56.256740 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:11:56.259922 21519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1: Bootstrap starting.
I20240503 14:11:56.263948 21519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:56.264925 21519 log.cc:824] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:56.267395 21519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1: No bootstrap required, opened a new log
I20240503 14:11:56.273945 21519 raft_consensus.cc:348] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21746e837d5438288989b67557befc1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43789 } }
I20240503 14:11:56.274396 21519 raft_consensus.cc:374] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:56.274544 21519 raft_consensus.cc:729] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21746e837d5438288989b67557befc1, State: Initialized, Role: FOLLOWER
I20240503 14:11:56.275147 21519 consensus_queue.cc:260] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21746e837d5438288989b67557befc1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43789 } }
I20240503 14:11:56.275329 21519 raft_consensus.cc:388] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:56.275461 21519 raft_consensus.cc:482] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:56.275600 21519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:56.278540 21519 raft_consensus.cc:504] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21746e837d5438288989b67557befc1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43789 } }
I20240503 14:11:56.278980 21519 leader_election.cc:302] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e21746e837d5438288989b67557befc1; no voters:
I20240503 14:11:56.279735 21519 leader_election.cc:288] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:56.279996 21524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:56.281651 21524 raft_consensus.cc:686] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [term 1 LEADER]: Becoming Leader. State: Replica: e21746e837d5438288989b67557befc1, State: Running, Role: LEADER
I20240503 14:11:56.282164 21519 sys_catalog.cc:564] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:56.282299 21524 consensus_queue.cc:237] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21746e837d5438288989b67557befc1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43789 } }
I20240503 14:11:56.286983 21525 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e21746e837d5438288989b67557befc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21746e837d5438288989b67557befc1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43789 } } }
I20240503 14:11:56.287283 21525 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:56.293470 21526 sys_catalog.cc:455] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e21746e837d5438288989b67557befc1. Latest consensus state: current_term: 1 leader_uuid: "e21746e837d5438288989b67557befc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e21746e837d5438288989b67557befc1" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43789 } } }
I20240503 14:11:56.293815 21526 sys_catalog.cc:458] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:56.294428 21531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:56.299278 21531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:11:56.307614 21531 catalog_manager.cc:1347] Generated new cluster ID: a57bab4d333f470f91faf32180cf8115
I20240503 14:11:56.307724 21531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:56.339663 21531 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:56.341228 21531 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:56.354665 21531 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e21746e837d5438288989b67557befc1: Generated new TSK 0
I20240503 14:11:56.355664 21531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:56.363265 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--builtin_ntp_servers=127.20.240.84:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 14:11:56.690493 21544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.690757 21544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.690867 21544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.698491 21544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.698652 21544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20240503 14:11:56.707363 21544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.708529 21544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.710132 21544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.722054 21550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.722801 21544 server_base.cc:1034] running on GCE node
W20240503 14:11:56.723356 21552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.724049 21549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.724606 21544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.725366 21544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.726577 21544 hybrid_clock.cc:648] HybridClock initialized: now 1714745516726515 us; error 48 us; skew 500 ppm
I20240503 14:11:56.726835 21544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.728826 21544 webserver.cc:460] Webserver started at http://127.20.240.65:45785/ using document root <none> and password file <none>
I20240503 14:11:56.729887 21544 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.730005 21544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.730309 21544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.732620 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-xhmt"
I20240503 14:11:56.733309 21544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-xhmt"
I20240503 14:11:56.737737 21544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:11:56.740851 21558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.742000 21544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:11:56.742151 21544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-xhmt"
I20240503 14:11:56.742336 21544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.757573 21544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.758759 21544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.758988 21544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.759768 21544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.761075 21544 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.761142 21544 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.761234 21544 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.761292 21544 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.791503 21544 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:33613
I20240503 14:11:56.791550 21670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:33613 every 8 connection(s)
I20240503 14:11:56.792819 21544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:11:56.798496 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 21544
I20240503 14:11:56.798684 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:11:56.801826 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--builtin_ntp_servers=127.20.240.84:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:56.805399 21671 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43789
I20240503 14:11:56.805680 21671 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:56.806438 21671 heartbeater.cc:511] Master 127.20.240.126:43789 requested a full tablet report, sending...
I20240503 14:11:56.808575 21484 ts_manager.cc:194] Registered new tserver with Master: b0ad0c5c7dbb47598f3ac5ce86f61ef7 (127.20.240.65:33613)
I20240503 14:11:56.810675 21484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:49663
W20240503 14:11:57.123477 21675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.123750 21675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.123854 21675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.131006 21675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.131165 21675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20240503 14:11:57.140204 21675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.141443 21675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.143059 21675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.155320 21680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.156170 21681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.156766 21683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.157665 21675 server_base.cc:1034] running on GCE node
I20240503 14:11:57.158319 21675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.159189 21675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.160399 21675 hybrid_clock.cc:648] HybridClock initialized: now 1714745517160345 us; error 38 us; skew 500 ppm
I20240503 14:11:57.160681 21675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.162822 21675 webserver.cc:460] Webserver started at http://127.20.240.66:33693/ using document root <none> and password file <none>
I20240503 14:11:57.163476 21675 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.163604 21675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.163918 21675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.167200 21675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "114ad6ddb6344cb0a8b06306761e6631"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-xhmt"
I20240503 14:11:57.167870 21675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "114ad6ddb6344cb0a8b06306761e6631"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-xhmt"
I20240503 14:11:57.172672 21675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:11:57.175873 21689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.177079 21675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:11:57.177227 21675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "114ad6ddb6344cb0a8b06306761e6631"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-xhmt"
I20240503 14:11:57.177385 21675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.189857 21675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.190629 21675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.190899 21675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.191772 21675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.193171 21675 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.193250 21675 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.193378 21675 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.193434 21675 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.226910 21675 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:44641
I20240503 14:11:57.226950 21801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:44641 every 8 connection(s)
I20240503 14:11:57.228600 21675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:11:57.229620 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 21675
I20240503 14:11:57.229787 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:11:57.233354 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--builtin_ntp_servers=127.20.240.84:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:57.242575 21802 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43789
I20240503 14:11:57.242830 21802 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.243538 21802 heartbeater.cc:511] Master 127.20.240.126:43789 requested a full tablet report, sending...
I20240503 14:11:57.245075 21484 ts_manager.cc:194] Registered new tserver with Master: 114ad6ddb6344cb0a8b06306761e6631 (127.20.240.66:44641)
I20240503 14:11:57.246254 21484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:38037
W20240503 14:11:57.556465 21805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.556763 21805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.556915 21805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.564196 21805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.564363 21805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20240503 14:11:57.573496 21805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.574822 21805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.576438 21805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.588814 21810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.589399 21811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.590039 21813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.590766 21805 server_base.cc:1034] running on GCE node
I20240503 14:11:57.591311 21805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.592185 21805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.593387 21805 hybrid_clock.cc:648] HybridClock initialized: now 1714745517593332 us; error 39 us; skew 500 ppm
I20240503 14:11:57.593674 21805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.595846 21805 webserver.cc:460] Webserver started at http://127.20.240.67:35835/ using document root <none> and password file <none>
I20240503 14:11:57.596525 21805 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.596648 21805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.596966 21805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.599511 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b12ce39e5e6c4c7f95fc30279f8ce7f0"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-xhmt"
I20240503 14:11:57.600284 21805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b12ce39e5e6c4c7f95fc30279f8ce7f0"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-xhmt"
I20240503 14:11:57.605373 21805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240503 14:11:57.608579 21819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.610342 21805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 14:11:57.610498 21805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b12ce39e5e6c4c7f95fc30279f8ce7f0"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-xhmt"
I20240503 14:11:57.610663 21805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.636018 21805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.636814 21805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.637560 21805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.638501 21805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.639918 21805 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.639997 21805 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.640128 21805 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.640190 21805 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.674043 21805 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:44139
I20240503 14:11:57.674098 21931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:44139 every 8 connection(s)
I20240503 14:11:57.675303 21805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:11:57.681164 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 21805
I20240503 14:11:57.681371 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:11:57.684515 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.68:0
--local_ip_for_outbound_sockets=127.20.240.68
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--builtin_ntp_servers=127.20.240.84:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:57.688287 21932 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43789
I20240503 14:11:57.688598 21932 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.689546 21932 heartbeater.cc:511] Master 127.20.240.126:43789 requested a full tablet report, sending...
I20240503 14:11:57.690971 21484 ts_manager.cc:194] Registered new tserver with Master: b12ce39e5e6c4c7f95fc30279f8ce7f0 (127.20.240.67:44139)
I20240503 14:11:57.691720 21484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:55339
I20240503 14:11:57.813978 21671 heartbeater.cc:503] Master 127.20.240.126:43789 was elected leader, sending a full tablet report...
W20240503 14:11:58.017288 21935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:58.017616 21935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:58.017760 21935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:58.025017 21935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:58.025236 21935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.68
I20240503 14:11:58.034365 21935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.68
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:58.035625 21935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:58.037307 21935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.050339 21940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.050798 21941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.051575 21943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.052287 21935 server_base.cc:1034] running on GCE node
I20240503 14:11:58.052824 21935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:58.053689 21935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:58.054913 21935 hybrid_clock.cc:648] HybridClock initialized: now 1714745518054844 us; error 31 us; skew 500 ppm
I20240503 14:11:58.055207 21935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.062952 21935 webserver.cc:460] Webserver started at http://127.20.240.68:38885/ using document root <none> and password file <none>
I20240503 14:11:58.063638 21935 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.063761 21935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.064049 21935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.066558 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f1f693f45a074cadb759f22fb453d236"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-xhmt"
I20240503 14:11:58.067243 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f1f693f45a074cadb759f22fb453d236"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-xhmt"
I20240503 14:11:58.072379 21935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20240503 14:11:58.075430 21949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.076565 21935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240503 14:11:58.076707 21935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f1f693f45a074cadb759f22fb453d236"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-xhmt"
I20240503 14:11:58.076885 21935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.091847 21935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.092573 21935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.092826 21935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.094105 21935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.095523 21935 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.095592 21935 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.095705 21935 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.095752 21935 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.129114 21935 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.68:40531
I20240503 14:11:58.129144 22061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.68:40531 every 8 connection(s)
I20240503 14:11:58.130369 21935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:11:58.132400 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 21935
I20240503 14:11:58.132586 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:11:58.136039 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:0
--local_ip_for_outbound_sockets=127.20.240.69
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--builtin_ntp_servers=127.20.240.84:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:58.147873 22062 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43789
I20240503 14:11:58.148147 22062 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.148880 22062 heartbeater.cc:511] Master 127.20.240.126:43789 requested a full tablet report, sending...
I20240503 14:11:58.150368 21484 ts_manager.cc:194] Registered new tserver with Master: f1f693f45a074cadb759f22fb453d236 (127.20.240.68:40531)
I20240503 14:11:58.151628 21484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.68:60499
I20240503 14:11:58.248991 21802 heartbeater.cc:503] Master 127.20.240.126:43789 was elected leader, sending a full tablet report...
W20240503 14:11:58.469141 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:58.469434 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:58.469573 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:58.477219 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:58.477437 22065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:11:58.486210 22065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43789
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:58.487514 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:58.489163 22065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:58.501472 22071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.502089 22070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:58.502981 22073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:58.503643 22065 server_base.cc:1034] running on GCE node
I20240503 14:11:58.504153 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:58.504956 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:58.506155 22065 hybrid_clock.cc:648] HybridClock initialized: now 1714745518506099 us; error 32 us; skew 500 ppm
I20240503 14:11:58.506449 22065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:58.508952 22065 webserver.cc:460] Webserver started at http://127.20.240.69:37213/ using document root <none> and password file <none>
I20240503 14:11:58.509616 22065 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:58.509732 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:58.510010 22065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:58.512396 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d834ac12c6444273a501d627be54e9e8"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-xhmt"
I20240503 14:11:58.513132 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d834ac12c6444273a501d627be54e9e8"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-xhmt"
I20240503 14:11:58.517768 22065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 14:11:58.520771 22079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.521983 22065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:11:58.522107 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d834ac12c6444273a501d627be54e9e8"
format_stamp: "Formatted at 2024-05-03 14:11:58 on dist-test-slave-xhmt"
I20240503 14:11:58.522274 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:58.538903 22065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:58.539654 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:58.539902 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:58.540786 22065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:58.542932 22065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:58.543022 22065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.543154 22065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:58.543218 22065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:58.578460 22065 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:33851
I20240503 14:11:58.578501 22191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:33851 every 8 connection(s)
I20240503 14:11:58.579690 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:11:58.584388 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 22065
I20240503 14:11:58.584607 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:11:58.594141 22192 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43789
I20240503 14:11:58.594444 22192 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:58.595194 22192 heartbeater.cc:511] Master 127.20.240.126:43789 requested a full tablet report, sending...
I20240503 14:11:58.596406 21484 ts_manager.cc:194] Registered new tserver with Master: d834ac12c6444273a501d627be54e9e8 (127.20.240.69:33851)
I20240503 14:11:58.597079 21484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:51733
I20240503 14:11:58.600867 21441 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:11:58.621316 21484 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34660:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:11:58.654140 21606 tablet_service.cc:1467] Processing CreateTablet for tablet 9441e1146f10499484ed682b5a083a02 (DEFAULT_TABLE table=TestTable [id=198a671b81614b29836b363c25ecf431]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:58.655797 21606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9441e1146f10499484ed682b5a083a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.655715 21737 tablet_service.cc:1467] Processing CreateTablet for tablet 9441e1146f10499484ed682b5a083a02 (DEFAULT_TABLE table=TestTable [id=198a671b81614b29836b363c25ecf431]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:58.657002 21737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9441e1146f10499484ed682b5a083a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.659694 22127 tablet_service.cc:1467] Processing CreateTablet for tablet 9441e1146f10499484ed682b5a083a02 (DEFAULT_TABLE table=TestTable [id=198a671b81614b29836b363c25ecf431]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:58.661055 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9441e1146f10499484ed682b5a083a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.667418 22210 tablet_bootstrap.cc:492] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631: Bootstrap starting.
I20240503 14:11:58.668998 22211 tablet_bootstrap.cc:492] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7: Bootstrap starting.
I20240503 14:11:58.671329 22210 tablet_bootstrap.cc:654] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.672487 22211 tablet_bootstrap.cc:654] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.672621 22210 log.cc:824] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:58.673714 22211 log.cc:824] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:58.679317 22212 tablet_bootstrap.cc:492] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8: Bootstrap starting.
I20240503 14:11:58.683837 22212 tablet_bootstrap.cc:654] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.684950 22212 log.cc:824] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:58.685065 22210 tablet_bootstrap.cc:492] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631: No bootstrap required, opened a new log
I20240503 14:11:58.685245 22211 tablet_bootstrap.cc:492] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7: No bootstrap required, opened a new log
I20240503 14:11:58.685304 22210 ts_tablet_manager.cc:1367] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.008s
I20240503 14:11:58.685541 22211 ts_tablet_manager.cc:1367] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240503 14:11:58.694115 22210 raft_consensus.cc:348] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:58.694499 21932 heartbeater.cc:503] Master 127.20.240.126:43789 was elected leader, sending a full tablet report...
I20240503 14:11:58.694525 22210 raft_consensus.cc:374] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.694612 22210 raft_consensus.cc:729] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 114ad6ddb6344cb0a8b06306761e6631, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.694636 22212 tablet_bootstrap.cc:492] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8: No bootstrap required, opened a new log
I20240503 14:11:58.694976 22212 ts_tablet_manager.cc:1367] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20240503 14:11:58.695235 22210 consensus_queue.cc:260] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:58.695225 22211 raft_consensus.cc:348] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:58.695647 22211 raft_consensus.cc:374] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.695739 22211 raft_consensus.cc:729] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ad0c5c7dbb47598f3ac5ce86f61ef7, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.696326 22211 consensus_queue.cc:260] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:58.697927 22210 ts_tablet_manager.cc:1398] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20240503 14:11:58.699378 22211 ts_tablet_manager.cc:1398] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7: Time spent starting tablet: real 0.014s user 0.004s sys 0.011s
I20240503 14:11:58.704742 22212 raft_consensus.cc:348] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:58.705024 22212 raft_consensus.cc:374] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.705116 22212 raft_consensus.cc:729] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d834ac12c6444273a501d627be54e9e8, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.705646 22212 consensus_queue.cc:260] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:58.706502 22192 heartbeater.cc:503] Master 127.20.240.126:43789 was elected leader, sending a full tablet report...
I20240503 14:11:58.707015 22212 ts_tablet_manager.cc:1398] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
W20240503 14:11:58.731148 21803 tablet.cc:2369] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:11:58.795711 21672 tablet.cc:2369] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:11:58.837097 22193 tablet.cc:2369] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:11:59.014613 22217 raft_consensus.cc:482] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:59.014909 22217 raft_consensus.cc:504] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:59.016680 22217 leader_election.cc:288] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d834ac12c6444273a501d627be54e9e8 (127.20.240.69:33851), 114ad6ddb6344cb0a8b06306761e6631 (127.20.240.66:44641)
I20240503 14:11:59.023454 22147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9441e1146f10499484ed682b5a083a02" candidate_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d834ac12c6444273a501d627be54e9e8" is_pre_election: true
I20240503 14:11:59.023530 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9441e1146f10499484ed682b5a083a02" candidate_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114ad6ddb6344cb0a8b06306761e6631" is_pre_election: true
I20240503 14:11:59.023913 22147 raft_consensus.cc:2454] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ad0c5c7dbb47598f3ac5ce86f61ef7 in term 0.
I20240503 14:11:59.023972 21757 raft_consensus.cc:2454] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0ad0c5c7dbb47598f3ac5ce86f61ef7 in term 0.
I20240503 14:11:59.024449 21562 leader_election.cc:302] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ad0c5c7dbb47598f3ac5ce86f61ef7, d834ac12c6444273a501d627be54e9e8; no voters:
I20240503 14:11:59.024782 22217 raft_consensus.cc:2781] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:59.024930 22217 raft_consensus.cc:482] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:59.025033 22217 raft_consensus.cc:3037] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.093717 22217 raft_consensus.cc:504] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:59.094710 22217 leader_election.cc:288] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [CANDIDATE]: Term 1 election: Requested vote from peers d834ac12c6444273a501d627be54e9e8 (127.20.240.69:33851), 114ad6ddb6344cb0a8b06306761e6631 (127.20.240.66:44641)
I20240503 14:11:59.095137 22147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9441e1146f10499484ed682b5a083a02" candidate_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d834ac12c6444273a501d627be54e9e8"
I20240503 14:11:59.095103 21757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9441e1146f10499484ed682b5a083a02" candidate_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "114ad6ddb6344cb0a8b06306761e6631"
I20240503 14:11:59.095345 21757 raft_consensus.cc:3037] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.095363 22147 raft_consensus.cc:3037] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:59.097455 22147 raft_consensus.cc:2454] T 9441e1146f10499484ed682b5a083a02 P d834ac12c6444273a501d627be54e9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ad0c5c7dbb47598f3ac5ce86f61ef7 in term 1.
I20240503 14:11:59.097891 21562 leader_election.cc:302] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0ad0c5c7dbb47598f3ac5ce86f61ef7, d834ac12c6444273a501d627be54e9e8; no voters:
I20240503 14:11:59.098169 22217 raft_consensus.cc:2781] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:59.098673 22217 raft_consensus.cc:686] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [term 1 LEADER]: Becoming Leader. State: Replica: b0ad0c5c7dbb47598f3ac5ce86f61ef7, State: Running, Role: LEADER
I20240503 14:11:59.099148 22217 consensus_queue.cc:237] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } }
I20240503 14:11:59.103695 21484 catalog_manager.cc:5484] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0ad0c5c7dbb47598f3ac5ce86f61ef7 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0ad0c5c7dbb47598f3ac5ce86f61ef7" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 } health_report { overall_health: UNKNOWN } } }
I20240503 14:11:59.154053 22062 heartbeater.cc:503] Master 127.20.240.126:43789 was elected leader, sending a full tablet report...
I20240503 14:11:59.190873 21757 raft_consensus.cc:2454] T 9441e1146f10499484ed682b5a083a02 P 114ad6ddb6344cb0a8b06306761e6631 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0ad0c5c7dbb47598f3ac5ce86f61ef7 in term 1.
I20240503 14:11:59.219362 21441 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:11:59.221784 21441 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b0ad0c5c7dbb47598f3ac5ce86f61ef7 to finish bootstrapping
I20240503 14:11:59.230613 21441 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 114ad6ddb6344cb0a8b06306761e6631 to finish bootstrapping
I20240503 14:11:59.237520 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b12ce39e5e6c4c7f95fc30279f8ce7f0 to finish bootstrapping
I20240503 14:11:59.246414 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f1f693f45a074cadb759f22fb453d236 to finish bootstrapping
I20240503 14:11:59.254971 21441 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d834ac12c6444273a501d627be54e9e8 to finish bootstrapping
I20240503 14:11:59.551522 22217 consensus_queue.cc:1035] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d834ac12c6444273a501d627be54e9e8" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:59.562273 22217 consensus_queue.cc:1035] T 9441e1146f10499484ed682b5a083a02 P b0ad0c5c7dbb47598f3ac5ce86f61ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "114ad6ddb6344cb0a8b06306761e6631" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 44641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:59.729218 21737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.729504 21606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:11:59.732872 22127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.733695 21867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.746565 21997 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.901350 21737 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.911900 21867 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.918388 21606 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:11:59.920775 21997 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.925873 22127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:00.068411 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 21544
I20240503 14:12:00.069618 21665 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.159648 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 21544
I20240503 14:12:00.184330 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 21675
I20240503 14:12:00.185328 21796 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.283447 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 21675
I20240503 14:12:00.301363 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 21805
I20240503 14:12:00.302315 21926 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.385753 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 21805
I20240503 14:12:00.402217 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 21935
I20240503 14:12:00.403146 22057 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.485908 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 21935
I20240503 14:12:00.502130 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 22065
I20240503 14:12:00.503109 22186 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.594518 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 22065
I20240503 14:12:00.611308 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 21453
I20240503 14:12:00.612320 21514 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:00.719658 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 21453
2024-05-03T14:12:00Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (4999 ms)
[----------] 1 test from AdminCliTest (4999 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240503 14:12:00.763433 21441 test_util.cc:268] Using random seed: 1302671875
I20240503 14:12:00.765152 21441 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:00.765228 21441 ts_itest-base.cc:113] --------------
I20240503 14:12:00.765265 21441 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:12:00.765316 21441 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:12:00.765357 21441 ts_itest-base.cc:116] --------------
2024-05-03T14:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:00Z Disabled control of system clock
I20240503 14:12:00.786957 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43983
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:43983
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 14:12:01.103601 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:01.103866 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:01.103994 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:01.111121 22318 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:01.111228 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:01.111297 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:01.111399 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:01.111459 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:01.119993 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:43983
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:01.121198 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:01.122853 22318 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:01.133631 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:01.134383 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:01.135102 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:01.136023 22318 server_base.cc:1034] running on GCE node
I20240503 14:12:01.136656 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:01.137920 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:01.139119 22318 hybrid_clock.cc:648] HybridClock initialized: now 1714745521139070 us; error 59 us; skew 500 ppm
I20240503 14:12:01.139478 22318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:01.141518 22318 webserver.cc:460] Webserver started at http://127.20.240.126:40883/ using document root <none> and password file <none>
I20240503 14:12:01.142316 22318 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:01.142438 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:01.142756 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:01.145140 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6311e551ab554d12bbaf274cfdea486a"
format_stamp: "Formatted at 2024-05-03 14:12:01 on dist-test-slave-xhmt"
I20240503 14:12:01.145853 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6311e551ab554d12bbaf274cfdea486a"
format_stamp: "Formatted at 2024-05-03 14:12:01 on dist-test-slave-xhmt"
I20240503 14:12:01.150542 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240503 14:12:01.153540 22332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:01.154752 22318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:01.154903 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6311e551ab554d12bbaf274cfdea486a"
format_stamp: "Formatted at 2024-05-03 14:12:01 on dist-test-slave-xhmt"
I20240503 14:12:01.155074 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:01.189934 22318 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:01.190955 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:01.191231 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:01.208474 22318 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:43983
I20240503 14:12:01.208494 22383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:43983 every 8 connection(s)
I20240503 14:12:01.209757 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:01.210747 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 22318
I20240503 14:12:01.210970 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:01.214452 22384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:01.226330 22384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a: Bootstrap starting.
I20240503 14:12:01.229168 22384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:01.230106 22384 log.cc:824] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:01.232362 22384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a: No bootstrap required, opened a new log
I20240503 14:12:01.237923 22384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6311e551ab554d12bbaf274cfdea486a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43983 } }
I20240503 14:12:01.238214 22384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:01.238329 22384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6311e551ab554d12bbaf274cfdea486a, State: Initialized, Role: FOLLOWER
I20240503 14:12:01.238839 22384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6311e551ab554d12bbaf274cfdea486a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43983 } }
I20240503 14:12:01.239017 22384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:01.239137 22384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:01.239274 22384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:01.271788 22384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6311e551ab554d12bbaf274cfdea486a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43983 } }
I20240503 14:12:01.272379 22384 leader_election.cc:302] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6311e551ab554d12bbaf274cfdea486a; no voters:
I20240503 14:12:01.273010 22384 leader_election.cc:288] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:01.273211 22389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:01.274410 22389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [term 1 LEADER]: Becoming Leader. State: Replica: 6311e551ab554d12bbaf274cfdea486a, State: Running, Role: LEADER
I20240503 14:12:01.275172 22384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:01.275095 22389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6311e551ab554d12bbaf274cfdea486a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43983 } }
I20240503 14:12:01.278462 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6311e551ab554d12bbaf274cfdea486a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6311e551ab554d12bbaf274cfdea486a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43983 } } }
I20240503 14:12:01.278843 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:01.278460 22391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6311e551ab554d12bbaf274cfdea486a. Latest consensus state: current_term: 1 leader_uuid: "6311e551ab554d12bbaf274cfdea486a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6311e551ab554d12bbaf274cfdea486a" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43983 } } }
I20240503 14:12:01.279194 22391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:01.280633 22397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:01.286120 22397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:01.295408 22397 catalog_manager.cc:1347] Generated new cluster ID: d91f3c760374484c80c05a6873a060f1
I20240503 14:12:01.295536 22397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:01.305016 22397 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:01.306188 22397 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:01.317618 22397 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 6311e551ab554d12bbaf274cfdea486a: Generated new TSK 0
I20240503 14:12:01.318763 22397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:01.336544 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--builtin_ntp_servers=127.20.240.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:12:01.654841 22408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:01.655114 22408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:01.655248 22408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:01.662449 22408 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:01.662568 22408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:01.662731 22408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20240503 14:12:01.671406 22408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:01.672574 22408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:01.674221 22408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:01.686275 22408 server_base.cc:1034] running on GCE node
W20240503 14:12:01.687088 22414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:01.687695 22416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:01.687795 22413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:01.688406 22408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:01.689186 22408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:01.690340 22408 hybrid_clock.cc:648] HybridClock initialized: now 1714745521690296 us; error 35 us; skew 500 ppm
I20240503 14:12:01.690644 22408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:01.692703 22408 webserver.cc:460] Webserver started at http://127.20.240.65:40943/ using document root <none> and password file <none>
I20240503 14:12:01.693377 22408 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:01.693485 22408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:01.693785 22408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:01.696260 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a551c8d93f4e4b39bd60520e864de474"
format_stamp: "Formatted at 2024-05-03 14:12:01 on dist-test-slave-xhmt"
I20240503 14:12:01.696955 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a551c8d93f4e4b39bd60520e864de474"
format_stamp: "Formatted at 2024-05-03 14:12:01 on dist-test-slave-xhmt"
I20240503 14:12:01.701478 22408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 14:12:01.704355 22422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:01.705426 22408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:01.705572 22408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a551c8d93f4e4b39bd60520e864de474"
format_stamp: "Formatted at 2024-05-03 14:12:01 on dist-test-slave-xhmt"
I20240503 14:12:01.705749 22408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:01.716974 22408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:01.717641 22408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:01.717860 22408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:01.718640 22408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:01.719938 22408 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:01.720031 22408 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:01.720134 22408 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:01.720204 22408 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:01.748907 22408 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:38979
I20240503 14:12:01.748962 22534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:38979 every 8 connection(s)
I20240503 14:12:01.750017 22408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:01.753579 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 22408
I20240503 14:12:01.753774 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:01.757017 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--builtin_ntp_servers=127.20.240.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:01.762522 22535 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43983
I20240503 14:12:01.762763 22535 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:01.763357 22535 heartbeater.cc:511] Master 127.20.240.126:43983 requested a full tablet report, sending...
I20240503 14:12:01.765388 22349 ts_manager.cc:194] Registered new tserver with Master: a551c8d93f4e4b39bd60520e864de474 (127.20.240.65:38979)
I20240503 14:12:01.767575 22349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:46335
W20240503 14:12:02.069988 22539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:02.070232 22539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:02.070339 22539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:02.077301 22539 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:02.077422 22539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:02.077533 22539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20240503 14:12:02.085549 22539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:02.086691 22539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:02.088228 22539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:02.100334 22547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:02.101006 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:02.101866 22545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:02.102674 22539 server_base.cc:1034] running on GCE node
I20240503 14:12:02.103266 22539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:02.104202 22539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:02.105415 22539 hybrid_clock.cc:648] HybridClock initialized: now 1714745522105381 us; error 43 us; skew 500 ppm
I20240503 14:12:02.105777 22539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:02.107931 22539 webserver.cc:460] Webserver started at http://127.20.240.66:34599/ using document root <none> and password file <none>
I20240503 14:12:02.108604 22539 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:02.108713 22539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:02.109040 22539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:02.111577 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "10e09633e1654d25a90496910627f8e6"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.112301 22539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "10e09633e1654d25a90496910627f8e6"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.116945 22539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 14:12:02.120093 22553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:02.121271 22539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:02.121417 22539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "10e09633e1654d25a90496910627f8e6"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.121599 22539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:02.142407 22539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:02.143121 22539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:02.143359 22539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:02.144280 22539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:02.145767 22539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:02.145964 22539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:02.146095 22539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:02.146178 22539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:02.175756 22539 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:37089
I20240503 14:12:02.175807 22665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:37089 every 8 connection(s)
I20240503 14:12:02.176970 22539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:02.184055 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 22539
I20240503 14:12:02.184223 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:02.186969 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--builtin_ntp_servers=127.20.240.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:02.189854 22666 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43983
I20240503 14:12:02.190109 22666 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:02.190848 22666 heartbeater.cc:511] Master 127.20.240.126:43983 requested a full tablet report, sending...
I20240503 14:12:02.192097 22349 ts_manager.cc:194] Registered new tserver with Master: 10e09633e1654d25a90496910627f8e6 (127.20.240.66:37089)
I20240503 14:12:02.193225 22349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:43873
W20240503 14:12:02.502743 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:02.503064 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:02.503202 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:02.510509 22669 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:02.510646 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:02.510810 22669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20240503 14:12:02.519584 22669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:02.520864 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:02.522599 22669 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:02.534812 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:02.535672 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:02.536309 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:02.537276 22669 server_base.cc:1034] running on GCE node
I20240503 14:12:02.537792 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:02.538633 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:02.539825 22669 hybrid_clock.cc:648] HybridClock initialized: now 1714745522539783 us; error 38 us; skew 500 ppm
I20240503 14:12:02.540185 22669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:02.543090 22669 webserver.cc:460] Webserver started at http://127.20.240.67:42197/ using document root <none> and password file <none>
I20240503 14:12:02.543903 22669 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:02.544018 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:02.544397 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:02.547648 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "40f0b8592ec44e11bda550e0708ed6e8"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.548390 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "40f0b8592ec44e11bda550e0708ed6e8"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.553292 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 14:12:02.556680 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:02.557958 22669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:02.558108 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "40f0b8592ec44e11bda550e0708ed6e8"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.558336 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:02.572508 22669 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:02.573328 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:02.573575 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:02.574527 22669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:02.575979 22669 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:02.576082 22669 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:02.576187 22669 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:02.576256 22669 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:02.606086 22669 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:34855
I20240503 14:12:02.606137 22795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:34855 every 8 connection(s)
I20240503 14:12:02.607381 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:02.614882 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 22669
I20240503 14:12:02.615027 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:02.618292 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.68:0
--local_ip_for_outbound_sockets=127.20.240.68
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--builtin_ntp_servers=127.20.240.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:02.619400 22796 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43983
I20240503 14:12:02.619720 22796 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:02.620370 22796 heartbeater.cc:511] Master 127.20.240.126:43983 requested a full tablet report, sending...
I20240503 14:12:02.621654 22349 ts_manager.cc:194] Registered new tserver with Master: 40f0b8592ec44e11bda550e0708ed6e8 (127.20.240.67:34855)
I20240503 14:12:02.622749 22349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:40881
I20240503 14:12:02.770339 22535 heartbeater.cc:503] Master 127.20.240.126:43983 was elected leader, sending a full tablet report...
W20240503 14:12:02.936115 22799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:02.936388 22799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:02.936506 22799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:02.943765 22799 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:02.943879 22799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:02.944038 22799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.68
I20240503 14:12:02.952533 22799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.68
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:02.953771 22799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:02.955734 22799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:02.969820 22804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:02.970474 22807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:02.972834 22805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:02.973775 22799 server_base.cc:1034] running on GCE node
I20240503 14:12:02.974490 22799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:02.975570 22799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:02.976862 22799 hybrid_clock.cc:648] HybridClock initialized: now 1714745522976784 us; error 64 us; skew 500 ppm
I20240503 14:12:02.977355 22799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:02.980382 22799 webserver.cc:460] Webserver started at http://127.20.240.68:46427/ using document root <none> and password file <none>
I20240503 14:12:02.981220 22799 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:02.981355 22799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:02.981710 22799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:02.984920 22799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e8135de545134a3698e0d118df397bdb"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.985853 22799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e8135de545134a3698e0d118df397bdb"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.992085 22799 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.005s
I20240503 14:12:02.996150 22813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:02.997756 22799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:02.997928 22799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e8135de545134a3698e0d118df397bdb"
format_stamp: "Formatted at 2024-05-03 14:12:02 on dist-test-slave-xhmt"
I20240503 14:12:02.998157 22799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:03.018466 22799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:03.019183 22799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:03.019415 22799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:03.020251 22799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:03.021557 22799 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:03.021641 22799 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:03.021745 22799 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:03.021811 22799 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:03.050209 22799 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.68:33627
I20240503 14:12:03.050287 22925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.68:33627 every 8 connection(s)
I20240503 14:12:03.051398 22799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:03.055430 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 22799
I20240503 14:12:03.055644 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:03.059005 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:0
--local_ip_for_outbound_sockets=127.20.240.69
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--builtin_ntp_servers=127.20.240.84:40675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:03.068811 22926 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43983
I20240503 14:12:03.069099 22926 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:03.069773 22926 heartbeater.cc:511] Master 127.20.240.126:43983 requested a full tablet report, sending...
I20240503 14:12:03.071085 22349 ts_manager.cc:194] Registered new tserver with Master: e8135de545134a3698e0d118df397bdb (127.20.240.68:33627)
I20240503 14:12:03.071820 22349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.68:57005
I20240503 14:12:03.195950 22666 heartbeater.cc:503] Master 127.20.240.126:43983 was elected leader, sending a full tablet report...
W20240503 14:12:03.375325 22929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:03.375619 22929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:03.375746 22929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:03.382756 22929 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:03.382879 22929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:03.383036 22929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:12:03.391280 22929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43983
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:03.392544 22929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:03.394208 22929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:03.405457 22934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:03.406288 22935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:03.406924 22937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:03.407825 22929 server_base.cc:1034] running on GCE node
I20240503 14:12:03.408349 22929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:03.409233 22929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:03.410408 22929 hybrid_clock.cc:648] HybridClock initialized: now 1714745523410356 us; error 51 us; skew 500 ppm
I20240503 14:12:03.410703 22929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:03.412750 22929 webserver.cc:460] Webserver started at http://127.20.240.69:37767/ using document root <none> and password file <none>
I20240503 14:12:03.413437 22929 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:03.413547 22929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:03.413843 22929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:03.416234 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ab11624e67fe4776838ab0c159d27e02"
format_stamp: "Formatted at 2024-05-03 14:12:03 on dist-test-slave-xhmt"
I20240503 14:12:03.416945 22929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ab11624e67fe4776838ab0c159d27e02"
format_stamp: "Formatted at 2024-05-03 14:12:03 on dist-test-slave-xhmt"
I20240503 14:12:03.421454 22929 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240503 14:12:03.424508 22943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:03.425720 22929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:03.425876 22929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ab11624e67fe4776838ab0c159d27e02"
format_stamp: "Formatted at 2024-05-03 14:12:03 on dist-test-slave-xhmt"
I20240503 14:12:03.426052 22929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:03.440920 22929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:03.441592 22929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:03.441843 22929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:03.442708 22929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:03.443913 22929 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:03.444000 22929 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:03.444095 22929 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:03.444159 22929 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:03.473516 22929 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:45953
I20240503 14:12:03.473564 23055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:45953 every 8 connection(s)
I20240503 14:12:03.474746 22929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:03.475208 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 22929
I20240503 14:12:03.475400 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:03.486582 23056 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43983
I20240503 14:12:03.486842 23056 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:03.487501 23056 heartbeater.cc:511] Master 127.20.240.126:43983 requested a full tablet report, sending...
I20240503 14:12:03.488615 22349 ts_manager.cc:194] Registered new tserver with Master: ab11624e67fe4776838ab0c159d27e02 (127.20.240.69:45953)
I20240503 14:12:03.489338 22349 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:54703
I20240503 14:12:03.491741 21441 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:03.505658 22349 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51714:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:03.533494 22731 tablet_service.cc:1467] Processing CreateTablet for tablet 66346ac50fd446c1917e8abbda6e8205 (DEFAULT_TABLE table=TestTable [id=14980b1734f747e4a23392b78b768663]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.534159 22601 tablet_service.cc:1467] Processing CreateTablet for tablet 66346ac50fd446c1917e8abbda6e8205 (DEFAULT_TABLE table=TestTable [id=14980b1734f747e4a23392b78b768663]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.535184 22731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66346ac50fd446c1917e8abbda6e8205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.535256 22601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66346ac50fd446c1917e8abbda6e8205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.536013 22861 tablet_service.cc:1467] Processing CreateTablet for tablet 66346ac50fd446c1917e8abbda6e8205 (DEFAULT_TABLE table=TestTable [id=14980b1734f747e4a23392b78b768663]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:03.537284 22861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66346ac50fd446c1917e8abbda6e8205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:03.545598 23074 tablet_bootstrap.cc:492] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6: Bootstrap starting.
I20240503 14:12:03.547622 23075 tablet_bootstrap.cc:492] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8: Bootstrap starting.
I20240503 14:12:03.549057 23074 tablet_bootstrap.cc:654] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.549158 23076 tablet_bootstrap.cc:492] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb: Bootstrap starting.
I20240503 14:12:03.550248 23074 log.cc:824] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:03.551043 23075 tablet_bootstrap.cc:654] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.552181 23076 tablet_bootstrap.cc:654] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:03.552233 23075 log.cc:824] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:03.553028 23074 tablet_bootstrap.cc:492] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6: No bootstrap required, opened a new log
I20240503 14:12:03.553282 23074 ts_tablet_manager.cc:1367] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:12:03.553297 23076 log.cc:824] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:03.555025 23075 tablet_bootstrap.cc:492] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8: No bootstrap required, opened a new log
I20240503 14:12:03.555287 23075 ts_tablet_manager.cc:1367] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 14:12:03.555440 23076 tablet_bootstrap.cc:492] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb: No bootstrap required, opened a new log
I20240503 14:12:03.555647 23076 ts_tablet_manager.cc:1367] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:12:03.561579 23076 raft_consensus.cc:348] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.561905 23076 raft_consensus.cc:374] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.561750 23074 raft_consensus.cc:348] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.562013 23076 raft_consensus.cc:729] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8135de545134a3698e0d118df397bdb, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.562160 23074 raft_consensus.cc:374] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.562358 23074 raft_consensus.cc:729] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10e09633e1654d25a90496910627f8e6, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.562726 23076 consensus_queue.cc:260] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.563025 23074 consensus_queue.cc:260] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.563882 22926 heartbeater.cc:503] Master 127.20.240.126:43983 was elected leader, sending a full tablet report...
I20240503 14:12:03.563606 23075 raft_consensus.cc:348] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.564141 23075 raft_consensus.cc:374] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:03.564244 23075 raft_consensus.cc:729] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f0b8592ec44e11bda550e0708ed6e8, State: Initialized, Role: FOLLOWER
I20240503 14:12:03.564389 23076 ts_tablet_manager.cc:1398] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20240503 14:12:03.565125 23075 consensus_queue.cc:260] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.566466 23074 ts_tablet_manager.cc:1398] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6: Time spent starting tablet: real 0.013s user 0.013s sys 0.002s
I20240503 14:12:03.567487 22796 heartbeater.cc:503] Master 127.20.240.126:43983 was elected leader, sending a full tablet report...
I20240503 14:12:03.567992 23075 ts_tablet_manager.cc:1398] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8: Time spent starting tablet: real 0.013s user 0.003s sys 0.010s
I20240503 14:12:03.580175 23081 raft_consensus.cc:482] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:03.580355 23081 raft_consensus.cc:504] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.581776 23081 leader_election.cc:288] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40f0b8592ec44e11bda550e0708ed6e8 (127.20.240.67:34855), e8135de545134a3698e0d118df397bdb (127.20.240.68:33627)
I20240503 14:12:03.590035 22751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66346ac50fd446c1917e8abbda6e8205" candidate_uuid: "10e09633e1654d25a90496910627f8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f0b8592ec44e11bda550e0708ed6e8" is_pre_election: true
I20240503 14:12:03.590066 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66346ac50fd446c1917e8abbda6e8205" candidate_uuid: "10e09633e1654d25a90496910627f8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8135de545134a3698e0d118df397bdb" is_pre_election: true
I20240503 14:12:03.590492 22881 raft_consensus.cc:2454] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e09633e1654d25a90496910627f8e6 in term 0.
I20240503 14:12:03.590495 22751 raft_consensus.cc:2454] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10e09633e1654d25a90496910627f8e6 in term 0.
I20240503 14:12:03.591022 22557 leader_election.cc:302] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e09633e1654d25a90496910627f8e6, e8135de545134a3698e0d118df397bdb; no voters:
I20240503 14:12:03.591341 23081 raft_consensus.cc:2781] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:03.591480 23081 raft_consensus.cc:482] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:03.591581 23081 raft_consensus.cc:3037] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.593480 23081 raft_consensus.cc:504] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.593972 23081 leader_election.cc:288] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [CANDIDATE]: Term 1 election: Requested vote from peers 40f0b8592ec44e11bda550e0708ed6e8 (127.20.240.67:34855), e8135de545134a3698e0d118df397bdb (127.20.240.68:33627)
I20240503 14:12:03.594334 22751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66346ac50fd446c1917e8abbda6e8205" candidate_uuid: "10e09633e1654d25a90496910627f8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40f0b8592ec44e11bda550e0708ed6e8"
I20240503 14:12:03.594429 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66346ac50fd446c1917e8abbda6e8205" candidate_uuid: "10e09633e1654d25a90496910627f8e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8135de545134a3698e0d118df397bdb"
I20240503 14:12:03.594506 22751 raft_consensus.cc:3037] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.594614 22881 raft_consensus.cc:3037] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:03.596572 22751 raft_consensus.cc:2454] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e09633e1654d25a90496910627f8e6 in term 1.
I20240503 14:12:03.596787 22881 raft_consensus.cc:2454] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10e09633e1654d25a90496910627f8e6 in term 1.
I20240503 14:12:03.596923 22554 leader_election.cc:302] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10e09633e1654d25a90496910627f8e6, 40f0b8592ec44e11bda550e0708ed6e8; no voters:
I20240503 14:12:03.597194 23081 raft_consensus.cc:2781] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:03.597586 23081 raft_consensus.cc:686] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [term 1 LEADER]: Becoming Leader. State: Replica: 10e09633e1654d25a90496910627f8e6, State: Running, Role: LEADER
I20240503 14:12:03.598055 23081 consensus_queue.cc:237] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } }
I20240503 14:12:03.602411 22347 catalog_manager.cc:5484] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10e09633e1654d25a90496910627f8e6 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "10e09633e1654d25a90496910627f8e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 } } peers { permanent_uuid: "10e09633e1654d25a90496910627f8e6" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37089 } } peers { permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 } } }
W20240503 14:12:03.609552 22797 tablet.cc:2369] T 66346ac50fd446c1917e8abbda6e8205 P 40f0b8592ec44e11bda550e0708ed6e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:03.623690 21441 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:03.625700 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a551c8d93f4e4b39bd60520e864de474 to finish bootstrapping
I20240503 14:12:03.634630 21441 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 10e09633e1654d25a90496910627f8e6 to finish bootstrapping
I20240503 14:12:03.641736 21441 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 40f0b8592ec44e11bda550e0708ed6e8 to finish bootstrapping
I20240503 14:12:03.648247 21441 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e8135de545134a3698e0d118df397bdb to finish bootstrapping
I20240503 14:12:03.654512 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ab11624e67fe4776838ab0c159d27e02 to finish bootstrapping
I20240503 14:12:03.662425 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 22408
I20240503 14:12:03.663336 22530 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:03.679356 22667 tablet.cc:2369] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:03.741070 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 22408
W20240503 14:12:03.803606 22927 tablet.cc:2369] T 66346ac50fd446c1917e8abbda6e8205 P e8135de545134a3698e0d118df397bdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:04.111773 23103 consensus_queue.cc:1035] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8135de545134a3698e0d118df397bdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.119870 23103 consensus_queue.cc:1035] T 66346ac50fd446c1917e8abbda6e8205 P 10e09633e1654d25a90496910627f8e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40f0b8592ec44e11bda550e0708ed6e8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:04.200958 22731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.218514 22601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:04.225704 22991 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.227291 22861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.347394 22601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:04.350677 22731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.353873 22861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.356251 22991 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.491727 23056 heartbeater.cc:503] Master 127.20.240.126:43983 was elected leader, sending a full tablet report...
I20240503 14:12:04.512806 22601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:04.512835 22731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.516117 22991 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.525529 22861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.101682 22731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.108755 22601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:05.109948 22991 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.112872 22861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.234481 22731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.235288 22601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:05.247650 22991 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.251766 22861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.397909 22731 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.402897 22601 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:05.414989 22991 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.420565 22861 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.563293 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 22539
I20240503 14:12:05.564658 22660 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:05.660955 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 22539
W20240503 14:12:05.679216 22335 connection.cc:535] server connection from 127.20.240.66:43873 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:12:05.679301 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 22669
I20240503 14:12:05.680266 22791 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:05.775192 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 22669
I20240503 14:12:05.792588 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 22799
I20240503 14:12:05.793565 22920 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:05.890636 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 22799
I20240503 14:12:05.908180 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 22929
I20240503 14:12:05.909121 23051 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:05.997972 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 22929
I20240503 14:12:06.014823 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 22318
I20240503 14:12:06.015888 22379 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.129278 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 22318
2024-05-03T14:12:06Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (5408 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5408 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240503 14:12:06.171911 21441 test_util.cc:268] Using random seed: 1308080369
I20240503 14:12:06.173846 21441 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:06.173920 21441 ts_itest-base.cc:113] --------------
I20240503 14:12:06.173976 21441 ts_itest-base.cc:114] 7 tablet servers
I20240503 14:12:06.174028 21441 ts_itest-base.cc:115] 1 replicas per TS
I20240503 14:12:06.174099 21441 ts_itest-base.cc:116] --------------
2024-05-03T14:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:06Z Disabled control of system clock
I20240503 14:12:06.196570 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43899
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:43899
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 14:12:06.504235 23295 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:12:06.504562 23295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:06.504671 23295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:06.504778 23295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:06.511868 23295 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:06.511987 23295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:06.512089 23295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:06.512156 23295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:06.512238 23295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:06.520742 23295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:43899
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:43899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:06.521940 23295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:06.523608 23295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:06.534276 23300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:06.535089 23301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:06.535821 23303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:06.536651 23295 server_base.cc:1034] running on GCE node
I20240503 14:12:06.537328 23295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:06.538654 23295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:06.539884 23295 hybrid_clock.cc:648] HybridClock initialized: now 1714745526539821 us; error 33 us; skew 500 ppm
I20240503 14:12:06.540299 23295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:06.542966 23295 webserver.cc:460] Webserver started at http://127.20.240.126:42595/ using document root <none> and password file <none>
I20240503 14:12:06.543921 23295 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:06.544085 23295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:06.544499 23295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:06.547514 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b"
format_stamp: "Formatted at 2024-05-03 14:12:06 on dist-test-slave-xhmt"
I20240503 14:12:06.548269 23295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b"
format_stamp: "Formatted at 2024-05-03 14:12:06 on dist-test-slave-xhmt"
I20240503 14:12:06.553411 23295 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:06.556785 23309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:06.558071 23295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:06.558231 23295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b"
format_stamp: "Formatted at 2024-05-03 14:12:06 on dist-test-slave-xhmt"
I20240503 14:12:06.558444 23295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:06.600916 23295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:06.601752 23295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:06.602030 23295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:06.618968 23295 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:43899
I20240503 14:12:06.619007 23360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:43899 every 8 connection(s)
I20240503 14:12:06.620237 23295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:06.621214 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 23295
I20240503 14:12:06.621438 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:06.624610 23361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:06.635447 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b: Bootstrap starting.
I20240503 14:12:06.638093 23361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:06.639008 23361 log.cc:824] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:06.641297 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b: No bootstrap required, opened a new log
I20240503 14:12:06.647169 23361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43899 } }
I20240503 14:12:06.647468 23361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:06.647558 23361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8d95e4965f4cb2bfdbc452fe5f7e0b, State: Initialized, Role: FOLLOWER
I20240503 14:12:06.648095 23361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43899 } }
I20240503 14:12:06.648308 23361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:06.648438 23361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:06.648617 23361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:06.650652 23361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43899 } }
I20240503 14:12:06.651067 23361 leader_election.cc:302] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c8d95e4965f4cb2bfdbc452fe5f7e0b; no voters:
I20240503 14:12:06.651543 23361 leader_election.cc:288] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:06.651649 23366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:06.652244 23366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [term 1 LEADER]: Becoming Leader. State: Replica: 4c8d95e4965f4cb2bfdbc452fe5f7e0b, State: Running, Role: LEADER
I20240503 14:12:06.652778 23366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43899 } }
I20240503 14:12:06.653357 23361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:06.655725 23367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43899 } } }
I20240503 14:12:06.656071 23367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:06.655978 23368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c8d95e4965f4cb2bfdbc452fe5f7e0b. Latest consensus state: current_term: 1 leader_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c8d95e4965f4cb2bfdbc452fe5f7e0b" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 43899 } } }
I20240503 14:12:06.656265 23368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:06.658974 23374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:06.663918 23374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:06.671108 23374 catalog_manager.cc:1347] Generated new cluster ID: 70e3d251d08e4136a74352a1e2d6862b
I20240503 14:12:06.671218 23374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:06.692940 23374 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:06.694083 23374 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:06.715724 23374 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4c8d95e4965f4cb2bfdbc452fe5f7e0b: Generated new TSK 0
I20240503 14:12:06.716533 23374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:06.727444 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:12:07.057139 23385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.057420 23385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:07.057513 23385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.064935 23385 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:07.065044 23385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:07.065165 23385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20240503 14:12:07.074782 23385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:07.076114 23385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:07.077780 23385 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.090162 23390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.090711 23391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.091571 23393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.092360 23385 server_base.cc:1034] running on GCE node
I20240503 14:12:07.092883 23385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:07.093755 23385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:07.094933 23385 hybrid_clock.cc:648] HybridClock initialized: now 1714745527094886 us; error 41 us; skew 500 ppm
I20240503 14:12:07.095213 23385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.097290 23385 webserver.cc:460] Webserver started at http://127.20.240.65:38157/ using document root <none> and password file <none>
I20240503 14:12:07.097983 23385 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.098091 23385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.098421 23385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.100797 23385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.101517 23385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.106349 23385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240503 14:12:07.109390 23399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.110586 23385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:07.110723 23385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.110893 23385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.128758 23385 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.129456 23385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:07.129707 23385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.130661 23385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:07.131955 23385 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:07.132050 23385 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.132150 23385 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:07.132220 23385 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.161379 23385 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:36725
I20240503 14:12:07.161428 23511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:36725 every 8 connection(s)
I20240503 14:12:07.162729 23385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:07.164819 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 23385
I20240503 14:12:07.165019 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:07.168357 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:07.176964 23512 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:07.177253 23512 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:07.177938 23512 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:07.180203 23326 ts_manager.cc:194] Registered new tserver with Master: 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:07.182581 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:50993
W20240503 14:12:07.499600 23516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.499883 23516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:07.499972 23516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.507617 23516 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:07.507738 23516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:07.507851 23516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20240503 14:12:07.517700 23516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:07.519026 23516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:07.520723 23516 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.532718 23524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.533519 23521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.534382 23522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.535223 23516 server_base.cc:1034] running on GCE node
I20240503 14:12:07.535766 23516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:07.536648 23516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:07.537854 23516 hybrid_clock.cc:648] HybridClock initialized: now 1714745527537798 us; error 50 us; skew 500 ppm
I20240503 14:12:07.538188 23516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.540411 23516 webserver.cc:460] Webserver started at http://127.20.240.66:40043/ using document root <none> and password file <none>
I20240503 14:12:07.541108 23516 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.541217 23516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.541548 23516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.544227 23516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7f3556100c8d46aaafb47e87defe8064"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.545012 23516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7f3556100c8d46aaafb47e87defe8064"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.550722 23516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20240503 14:12:07.553921 23530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.555327 23516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:07.555469 23516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7f3556100c8d46aaafb47e87defe8064"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.555644 23516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.578347 23516 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.579186 23516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:07.579470 23516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.581599 23516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:07.583109 23516 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:07.583205 23516 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.583331 23516 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:07.583405 23516 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.614770 23516 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:36169
I20240503 14:12:07.614812 23642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:36169 every 8 connection(s)
I20240503 14:12:07.616107 23516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:07.617174 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 23516
I20240503 14:12:07.617344 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:07.620855 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:07.632211 23643 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:07.632545 23643 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:07.633544 23643 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:07.635231 23326 ts_manager.cc:194] Registered new tserver with Master: 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:07.635977 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:42955
W20240503 14:12:07.941335 23646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.941643 23646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:07.941802 23646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.949084 23646 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:07.949208 23646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:07.949364 23646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20240503 14:12:07.958026 23646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:07.959319 23646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:07.961055 23646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.973021 23654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.973809 23651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.974555 23652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.975375 23646 server_base.cc:1034] running on GCE node
I20240503 14:12:07.975873 23646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:07.976716 23646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:07.977903 23646 hybrid_clock.cc:648] HybridClock initialized: now 1714745527977850 us; error 44 us; skew 500 ppm
I20240503 14:12:07.978168 23646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.980213 23646 webserver.cc:460] Webserver started at http://127.20.240.67:44429/ using document root <none> and password file <none>
I20240503 14:12:07.980857 23646 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.980947 23646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.981212 23646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.983615 23646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1a4466679fbc43618c8fb2e3d8a26dfd"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.984305 23646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1a4466679fbc43618c8fb2e3d8a26dfd"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.988883 23646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:12:07.991927 23660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.993124 23646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:07.993263 23646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1a4466679fbc43618c8fb2e3d8a26dfd"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:07.993435 23646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.010169 23646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.010972 23646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:08.011229 23646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.012081 23646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.013447 23646 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.013535 23646 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.013645 23646 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.013718 23646 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.043346 23646 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:41087
I20240503 14:12:08.043403 23772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:41087 every 8 connection(s)
I20240503 14:12:08.044571 23646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:08.049788 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 23646
I20240503 14:12:08.049965 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:08.052769 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.68:0
--local_ip_for_outbound_sockets=127.20.240.68
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:08.057149 23773 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:08.057415 23773 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.058107 23773 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:08.059456 23326 ts_manager.cc:194] Registered new tserver with Master: 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67:41087)
I20240503 14:12:08.060230 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:60013
I20240503 14:12:08.185487 23512 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
W20240503 14:12:08.367126 23776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.367414 23776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:08.367550 23776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.375031 23776 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:08.375162 23776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:08.375309 23776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.68
I20240503 14:12:08.384080 23776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.68
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:08.385362 23776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:08.387089 23776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:08.399111 23781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.399884 23782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.400594 23784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.401394 23776 server_base.cc:1034] running on GCE node
I20240503 14:12:08.401929 23776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:08.402807 23776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:08.403986 23776 hybrid_clock.cc:648] HybridClock initialized: now 1714745528403950 us; error 35 us; skew 500 ppm
I20240503 14:12:08.404268 23776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:08.411442 23776 webserver.cc:460] Webserver started at http://127.20.240.68:42509/ using document root <none> and password file <none>
I20240503 14:12:08.412133 23776 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:08.412240 23776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:08.412534 23776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:08.415015 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2062470ed9164d389aa2e60973ecdcdb"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:08.415743 23776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2062470ed9164d389aa2e60973ecdcdb"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:08.420449 23776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240503 14:12:08.423481 23790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.424672 23776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:08.424813 23776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2062470ed9164d389aa2e60973ecdcdb"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:08.424983 23776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.450971 23776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.451735 23776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:08.451987 23776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.452847 23776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.454228 23776 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.454335 23776 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.454437 23776 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.454507 23776 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.485895 23776 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.68:44039
I20240503 14:12:08.485958 23902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.68:44039 every 8 connection(s)
I20240503 14:12:08.487311 23776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:08.489974 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 23776
I20240503 14:12:08.490139 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:08.493430 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:0
--local_ip_for_outbound_sockets=127.20.240.69
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:08.500702 23903 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:08.500959 23903 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.501603 23903 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:08.502839 23326 ts_manager.cc:194] Registered new tserver with Master: 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68:44039)
I20240503 14:12:08.503557 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.68:36941
I20240503 14:12:08.639040 23643 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
W20240503 14:12:08.820272 23906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.820539 23906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:08.820624 23906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.827554 23906 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:08.827673 23906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:08.827780 23906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:12:08.836262 23906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:08.837970 23906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:08.839692 23906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:08.852759 23911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.853363 23912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.853933 23914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.854689 23906 server_base.cc:1034] running on GCE node
I20240503 14:12:08.855235 23906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:08.856124 23906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:08.857349 23906 hybrid_clock.cc:648] HybridClock initialized: now 1714745528857303 us; error 38 us; skew 500 ppm
I20240503 14:12:08.857656 23906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:08.859869 23906 webserver.cc:460] Webserver started at http://127.20.240.69:35873/ using document root <none> and password file <none>
I20240503 14:12:08.860580 23906 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:08.860692 23906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:08.860980 23906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:08.863593 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4fbbe1e5ec864182b3116ef1278f7a3a"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:08.864336 23906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4fbbe1e5ec864182b3116ef1278f7a3a"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:08.869058 23906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:12:08.872243 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.873486 23906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:08.873623 23906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4fbbe1e5ec864182b3116ef1278f7a3a"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:08.873802 23906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.901517 23906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.902361 23906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:08.902619 23906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.903507 23906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.904891 23906 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.904986 23906 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.905097 23906 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.905167 23906 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.935420 23906 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:38795
I20240503 14:12:08.935482 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:38795 every 8 connection(s)
I20240503 14:12:08.936650 23906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:08.942528 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 23906
I20240503 14:12:08.942726 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:08.945734 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.70:0
--local_ip_for_outbound_sockets=127.20.240.70
--webserver_interface=127.20.240.70
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:08.948765 24033 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:08.949095 24033 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.949993 24033 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:08.951364 23326 ts_manager.cc:194] Registered new tserver with Master: 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69:38795)
I20240503 14:12:08.952075 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:60609
I20240503 14:12:09.062414 23773 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
W20240503 14:12:09.258153 24036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:09.258460 24036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:09.258594 24036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:09.265834 24036 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:09.265954 24036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:09.266093 24036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.70
I20240503 14:12:09.274701 24036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.240.70
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.70
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:09.275960 24036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:09.277653 24036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:09.288888 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.289425 24041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.290380 24044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:09.291218 24036 server_base.cc:1034] running on GCE node
I20240503 14:12:09.291730 24036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:09.292533 24036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:09.293730 24036 hybrid_clock.cc:648] HybridClock initialized: now 1714745529293692 us; error 37 us; skew 500 ppm
I20240503 14:12:09.294015 24036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:09.295985 24036 webserver.cc:460] Webserver started at http://127.20.240.70:35929/ using document root <none> and password file <none>
I20240503 14:12:09.296686 24036 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:09.296811 24036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:09.297106 24036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:09.299638 24036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "95dfe9d1f79f4a8696fc5586e3ea3731"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:09.300369 24036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "95dfe9d1f79f4a8696fc5586e3ea3731"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:09.305070 24036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240503 14:12:09.308058 24050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.309201 24036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:09.309341 24036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "95dfe9d1f79f4a8696fc5586e3ea3731"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:09.309509 24036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:09.330719 24036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:09.331391 24036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:09.331631 24036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:09.332465 24036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:09.333811 24036 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:09.333912 24036 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.334015 24036 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:09.334093 24036 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.363895 24036 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.70:34091
I20240503 14:12:09.363946 24162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.70:34091 every 8 connection(s)
I20240503 14:12:09.365098 24036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:09.372674 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 24036
I20240503 14:12:09.372825 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:12:09.375999 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.71:0
--local_ip_for_outbound_sockets=127.20.240.71
--webserver_interface=127.20.240.71
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:09.377290 24163 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:09.377651 24163 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:09.378569 24163 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:09.379964 23326 ts_manager.cc:194] Registered new tserver with Master: 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70:34091)
I20240503 14:12:09.380692 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.70:50635
I20240503 14:12:09.505802 23903 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
W20240503 14:12:09.695045 24166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:09.695313 24166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:09.695437 24166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:09.702720 24166 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:09.702833 24166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:09.702982 24166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.71
I20240503 14:12:09.712128 24166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.240.71
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.71
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:09.713438 24166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:09.715183 24166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:09.727795 24171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.728394 24174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.729308 24172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:09.730147 24166 server_base.cc:1034] running on GCE node
I20240503 14:12:09.730726 24166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:09.731609 24166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:09.732797 24166 hybrid_clock.cc:648] HybridClock initialized: now 1714745529732761 us; error 31 us; skew 500 ppm
I20240503 14:12:09.733075 24166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:09.735199 24166 webserver.cc:460] Webserver started at http://127.20.240.71:38509/ using document root <none> and password file <none>
I20240503 14:12:09.735865 24166 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:09.735971 24166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:09.736258 24166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:09.738608 24166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "0e71aaeb321d4364bc1d81d64815b201"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:09.739307 24166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "0e71aaeb321d4364bc1d81d64815b201"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:09.743760 24166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:09.746750 24180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.747872 24166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:09.748014 24166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0e71aaeb321d4364bc1d81d64815b201"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:09.748191 24166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:09.766094 24166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:09.766867 24166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:09.767138 24166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:09.768013 24166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:09.769825 24166 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:09.769922 24166 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.770027 24166 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:09.770110 24166 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.805132 24166 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.71:43179
I20240503 14:12:09.805179 24292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.71:43179 every 8 connection(s)
I20240503 14:12:09.806409 24166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:09.813246 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 24166
I20240503 14:12:09.813601 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:12:09.821221 24293 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:09.821503 24293 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:09.822171 24293 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:09.823330 23326 ts_manager.cc:194] Registered new tserver with Master: 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71:43179)
I20240503 14:12:09.824012 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.71:49897
I20240503 14:12:09.829605 21441 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:09.844962 23326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60850:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:09.871229 23838 tablet_service.cc:1467] Processing CreateTablet for tablet db65a7aa1497429cb05bcd6b94d88914 (DEFAULT_TABLE table=TestTable [id=002b854d9ad34759bd833ba02ff8626f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.872450 23838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db65a7aa1497429cb05bcd6b94d88914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.882429 24307 tablet_bootstrap.cc:492] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Bootstrap starting.
I20240503 14:12:09.885531 24307 tablet_bootstrap.cc:654] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.886525 24307 log.cc:824] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:09.888644 24307 tablet_bootstrap.cc:492] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: No bootstrap required, opened a new log
I20240503 14:12:09.888861 24307 ts_tablet_manager.cc:1367] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 14:12:09.894480 24307 raft_consensus.cc:348] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:09.894773 24307 raft_consensus.cc:374] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.894862 24307 raft_consensus.cc:729] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2062470ed9164d389aa2e60973ecdcdb, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.895397 24307 consensus_queue.cc:260] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:09.895587 24307 raft_consensus.cc:388] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:09.896158 24307 raft_consensus.cc:482] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:09.896378 24307 raft_consensus.cc:3037] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:09.898557 24307 raft_consensus.cc:504] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:09.899101 24307 leader_election.cc:302] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2062470ed9164d389aa2e60973ecdcdb; no voters:
I20240503 14:12:09.899673 24307 leader_election.cc:288] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:09.899789 24309 raft_consensus.cc:2781] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:09.900382 24309 raft_consensus.cc:686] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 LEADER]: Becoming Leader. State: Replica: 2062470ed9164d389aa2e60973ecdcdb, State: Running, Role: LEADER
I20240503 14:12:09.900954 24309 consensus_queue.cc:237] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:09.901402 24307 ts_tablet_manager.cc:1398] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20240503 14:12:09.905962 23326 catalog_manager.cc:5484] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68). New cstate: current_term: 1 leader_uuid: "2062470ed9164d389aa2e60973ecdcdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:09.915273 21441 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:09.917003 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5ee9b8f29d3349f4ba05425d5287ff58 to finish bootstrapping
I20240503 14:12:09.925230 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7f3556100c8d46aaafb47e87defe8064 to finish bootstrapping
I20240503 14:12:09.933605 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1a4466679fbc43618c8fb2e3d8a26dfd to finish bootstrapping
I20240503 14:12:09.942034 21441 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2062470ed9164d389aa2e60973ecdcdb to finish bootstrapping
I20240503 14:12:09.949366 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4fbbe1e5ec864182b3116ef1278f7a3a to finish bootstrapping
I20240503 14:12:09.954845 24033 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
I20240503 14:12:09.957190 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 95dfe9d1f79f4a8696fc5586e3ea3731 to finish bootstrapping
I20240503 14:12:09.965312 21441 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0e71aaeb321d4364bc1d81d64815b201 to finish bootstrapping
I20240503 14:12:09.972738 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 23646
I20240503 14:12:09.973639 23768 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:10.051766 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 23646
I20240503 14:12:10.067355 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 23776
I20240503 14:12:10.068439 23898 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:10.154676 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 23776
I20240503 14:12:10.171504 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 23906
I20240503 14:12:10.172580 24028 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:10.253819 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 23906
I20240503 14:12:10.269695 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 24036
I20240503 14:12:10.270776 24158 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:10.349298 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 24036
I20240503 14:12:10.364671 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 24166
I20240503 14:12:10.365679 24288 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:10.443775 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 24166
I20240503 14:12:14.211109 23325 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60850:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:14.229444 23447 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3bf5d7cae7411486038ac34ab035e4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9694e7b92ac404db0c54d5ee5dfc86d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.229439 23578 tablet_service.cc:1467] Processing CreateTablet for tablet 2168c4283b0b4d63b6bb05fd20a41c9f (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9694e7b92ac404db0c54d5ee5dfc86d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.229493 23446 tablet_service.cc:1467] Processing CreateTablet for tablet 3aba18dc398d4c8eb1dfce74850a1478 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d9694e7b92ac404db0c54d5ee5dfc86d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:14.230816 23578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2168c4283b0b4d63b6bb05fd20a41c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.230840 23446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aba18dc398d4c8eb1dfce74850a1478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.231230 23447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3bf5d7cae7411486038ac34ab035e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:14.241966 24334 tablet_bootstrap.cc:492] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:14.243326 24335 tablet_bootstrap.cc:492] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064: Bootstrap starting.
I20240503 14:12:14.244989 24334 tablet_bootstrap.cc:654] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.245756 24335 tablet_bootstrap.cc:654] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.245935 24334 log.cc:824] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:14.246870 24335 log.cc:824] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:14.248265 24334 tablet_bootstrap.cc:492] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:14.248486 24334 ts_tablet_manager.cc:1367] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 14:12:14.248989 24335 tablet_bootstrap.cc:492] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064: No bootstrap required, opened a new log
I20240503 14:12:14.249197 24335 ts_tablet_manager.cc:1367] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:14.254467 24334 raft_consensus.cc:348] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.254755 24334 raft_consensus.cc:374] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.254880 24334 raft_consensus.cc:729] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.254937 24335 raft_consensus.cc:348] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:14.255280 24335 raft_consensus.cc:374] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.255373 24335 raft_consensus.cc:729] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.255529 24334 consensus_queue.cc:260] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.255760 24334 raft_consensus.cc:388] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:14.255882 24334 raft_consensus.cc:482] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:14.255910 24335 consensus_queue.cc:260] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:14.256024 24334 raft_consensus.cc:3037] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.256119 24335 raft_consensus.cc:388] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:14.256235 24335 raft_consensus.cc:482] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:14.256373 24335 raft_consensus.cc:3037] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.257920 24334 raft_consensus.cc:504] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.258159 24335 raft_consensus.cc:504] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:14.258421 24334 leader_election.cc:302] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:14.258615 24335 leader_election.cc:302] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:14.259333 24334 leader_election.cc:288] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:14.259516 24338 raft_consensus.cc:2781] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.260496 24335 leader_election.cc:288] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:14.260603 24339 raft_consensus.cc:2781] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.264904 24339 raft_consensus.cc:686] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:14.265404 24339 consensus_queue.cc:237] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:14.265939 24335 ts_tablet_manager.cc:1398] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20240503 14:12:14.266736 24338 raft_consensus.cc:686] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:14.267443 24334 ts_tablet_manager.cc:1398] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20240503 14:12:14.267297 24338 consensus_queue.cc:237] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.267796 24334 tablet_bootstrap.cc:492] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:14.269366 23325 catalog_manager.cc:5484] T 2168c4283b0b4d63b6bb05fd20a41c9f P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } }
I20240503 14:12:14.269646 24334 tablet_bootstrap.cc:654] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:14.275996 24334 tablet_bootstrap.cc:492] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:14.276154 24334 ts_tablet_manager.cc:1367] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:12:14.277210 24334 raft_consensus.cc:348] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.277405 24334 raft_consensus.cc:374] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:14.277520 24334 raft_consensus.cc:729] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:14.277813 24334 consensus_queue.cc:260] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.278059 24334 raft_consensus.cc:388] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:14.278182 24334 raft_consensus.cc:482] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:14.278323 24334 raft_consensus.cc:3037] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:14.278322 23325 catalog_manager.cc:5484] T 1e3bf5d7cae7411486038ac34ab035e4 P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
I20240503 14:12:14.280402 24334 raft_consensus.cc:504] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.280598 24334 leader_election.cc:302] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:14.280797 24334 leader_election.cc:288] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:14.280844 24338 raft_consensus.cc:2781] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:14.281069 24338 raft_consensus.cc:686] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:14.281272 24334 ts_tablet_manager.cc:1398] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240503 14:12:14.281334 24338 consensus_queue.cc:237] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:14.283119 23325 catalog_manager.cc:5484] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
W20240503 14:12:14.611496 24350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:14.611801 24350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:14.619254 24350 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 134.97 ms
time per row: 0.0674848 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:15.807967 23326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60850:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:15.816433 23578 tablet_service.cc:1467] Processing CreateTablet for tablet 54ff6aaa60734f548ff2b8da41ea5b3a (DEFAULT_TABLE table=rebalance_test_table_1 [id=78b403f6ecf94c72ada7ac48f4cc6957]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.816452 23577 tablet_service.cc:1467] Processing CreateTablet for tablet 4580e6da50584b93a7eab07537fa8a73 (DEFAULT_TABLE table=rebalance_test_table_1 [id=78b403f6ecf94c72ada7ac48f4cc6957]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.816797 23446 tablet_service.cc:1467] Processing CreateTablet for tablet d7e3d00a314845b1959d4538b4c0a05e (DEFAULT_TABLE table=rebalance_test_table_1 [id=78b403f6ecf94c72ada7ac48f4cc6957]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:15.816952 23578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ff6aaa60734f548ff2b8da41ea5b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.817308 23446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7e3d00a314845b1959d4538b4c0a05e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.817322 23577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4580e6da50584b93a7eab07537fa8a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:15.824050 24395 tablet_bootstrap.cc:492] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: Bootstrap starting.
I20240503 14:12:15.825225 24396 tablet_bootstrap.cc:492] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:15.825639 24395 tablet_bootstrap.cc:654] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.826891 24396 tablet_bootstrap.cc:654] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.827490 24395 tablet_bootstrap.cc:492] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: No bootstrap required, opened a new log
I20240503 14:12:15.827620 24395 ts_tablet_manager.cc:1367] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:15.828599 24396 tablet_bootstrap.cc:492] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:15.828521 24395 raft_consensus.cc:348] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.828678 24395 raft_consensus.cc:374] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.828727 24396 ts_tablet_manager.cc:1367] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:15.828775 24395 raft_consensus.cc:729] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.829013 24395 consensus_queue.cc:260] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.829196 24395 raft_consensus.cc:388] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:15.829294 24395 raft_consensus.cc:482] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:15.829404 24395 raft_consensus.cc:3037] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.829463 24396 raft_consensus.cc:348] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:15.829620 24396 raft_consensus.cc:374] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.829700 24396 raft_consensus.cc:729] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.829919 24396 consensus_queue.cc:260] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:15.830185 24396 raft_consensus.cc:388] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:15.830298 24396 raft_consensus.cc:482] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:15.830405 24396 raft_consensus.cc:3037] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.831966 24395 raft_consensus.cc:504] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.832191 24395 leader_election.cc:302] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:15.832559 24395 leader_election.cc:288] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:15.832700 24399 raft_consensus.cc:2781] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.832875 24396 raft_consensus.cc:504] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:15.833184 24395 ts_tablet_manager.cc:1398] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:15.833149 24396 leader_election.cc:302] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:15.833541 24395 tablet_bootstrap.cc:492] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: Bootstrap starting.
I20240503 14:12:15.835705 24395 tablet_bootstrap.cc:654] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:15.836920 24396 leader_election.cc:288] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:15.837040 24402 raft_consensus.cc:2781] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.838151 24395 tablet_bootstrap.cc:492] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: No bootstrap required, opened a new log
I20240503 14:12:15.838434 24395 ts_tablet_manager.cc:1367] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:15.839361 24399 raft_consensus.cc:686] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:15.839399 24395 raft_consensus.cc:348] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.839555 24395 raft_consensus.cc:374] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:15.839641 24395 raft_consensus.cc:729] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Initialized, Role: FOLLOWER
I20240503 14:12:15.839866 24395 consensus_queue.cc:260] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.839890 24399 consensus_queue.cc:237] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.841043 24395 raft_consensus.cc:388] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:15.841166 24395 raft_consensus.cc:482] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:15.841203 24396 ts_tablet_manager.cc:1398] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.012s user 0.003s sys 0.009s
I20240503 14:12:15.841267 24395 raft_consensus.cc:3037] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:15.841722 24402 raft_consensus.cc:686] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:15.842073 24402 consensus_queue.cc:237] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:15.843408 23326 catalog_manager.cc:5484] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } }
I20240503 14:12:15.844405 24395 raft_consensus.cc:504] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.844578 24395 leader_election.cc:302] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:15.844763 24395 leader_election.cc:288] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:15.844961 24400 raft_consensus.cc:2781] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:15.850538 24400 raft_consensus.cc:686] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:15.850914 24400 consensus_queue.cc:237] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:15.851608 24395 ts_tablet_manager.cc:1398] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240503 14:12:15.853191 23326 catalog_manager.cc:5484] T d7e3d00a314845b1959d4538b4c0a05e P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
I20240503 14:12:15.853549 23317 catalog_manager.cc:5484] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } }
W20240503 14:12:16.176756 24413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.177086 24413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:16.184243 24413 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 107.075 ms
time per row: 0.0535376 ms
I20240503 14:12:16.425664 23317 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60850:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:16.433431 23446 tablet_service.cc:1467] Processing CreateTablet for tablet 4b33bbffc8d1476e91ac19f35c820305 (DEFAULT_TABLE table=rebalance_test_table_2 [id=13c93033066d40228c6279535bf62553]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.433588 23577 tablet_service.cc:1467] Processing CreateTablet for tablet a71476f51de74a25ba84fc92ef0f9414 (DEFAULT_TABLE table=rebalance_test_table_2 [id=13c93033066d40228c6279535bf62553]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.433822 23447 tablet_service.cc:1467] Processing CreateTablet for tablet d433f9489b204c8f9e1ef5c2164c5a6e (DEFAULT_TABLE table=rebalance_test_table_2 [id=13c93033066d40228c6279535bf62553]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.433934 23446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b33bbffc8d1476e91ac19f35c820305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.433995 23577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71476f51de74a25ba84fc92ef0f9414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.434180 23447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d433f9489b204c8f9e1ef5c2164c5a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.440030 24454 tablet_bootstrap.cc:492] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:16.440876 24455 tablet_bootstrap.cc:492] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: Bootstrap starting.
I20240503 14:12:16.441408 24454 tablet_bootstrap.cc:654] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.442071 24455 tablet_bootstrap.cc:654] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.443138 24454 tablet_bootstrap.cc:492] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:16.443248 24454 ts_tablet_manager.cc:1367] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:16.443677 24455 tablet_bootstrap.cc:492] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: No bootstrap required, opened a new log
I20240503 14:12:16.443802 24455 ts_tablet_manager.cc:1367] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:16.443805 24454 raft_consensus.cc:348] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.443989 24454 raft_consensus.cc:374] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.444089 24454 raft_consensus.cc:729] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.444305 24454 consensus_queue.cc:260] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.444458 24454 raft_consensus.cc:388] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:16.444535 24454 raft_consensus.cc:482] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:16.444478 24455 raft_consensus.cc:348] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:16.444643 24454 raft_consensus.cc:3037] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.444661 24455 raft_consensus.cc:374] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.444744 24455 raft_consensus.cc:729] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.444957 24455 consensus_queue.cc:260] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:16.445107 24455 raft_consensus.cc:388] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:16.445191 24455 raft_consensus.cc:482] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:16.445294 24455 raft_consensus.cc:3037] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.446624 24454 raft_consensus.cc:504] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.446856 24454 leader_election.cc:302] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:16.447055 24455 raft_consensus.cc:504] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:16.447250 24454 leader_election.cc:288] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:16.447319 24405 raft_consensus.cc:2781] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.447305 24455 leader_election.cc:302] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:16.447490 24455 leader_election.cc:288] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:16.447548 24400 raft_consensus.cc:2781] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.447912 24400 raft_consensus.cc:686] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:16.448035 24455 ts_tablet_manager.cc:1398] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:16.448196 24400 consensus_queue.cc:237] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:16.448606 24405 raft_consensus.cc:686] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:16.448951 24454 ts_tablet_manager.cc:1398] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:16.448985 24405 consensus_queue.cc:237] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.449164 24454 tablet_bootstrap.cc:492] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:16.451191 23317 catalog_manager.cc:5484] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } }
I20240503 14:12:16.451635 24454 tablet_bootstrap.cc:654] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.452769 23325 catalog_manager.cc:5484] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
I20240503 14:12:16.454537 24454 tablet_bootstrap.cc:492] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:16.454680 24454 ts_tablet_manager.cc:1367] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:16.455420 24454 raft_consensus.cc:348] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.455636 24454 raft_consensus.cc:374] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.455726 24454 raft_consensus.cc:729] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.455947 24454 consensus_queue.cc:260] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.456121 24454 raft_consensus.cc:388] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:16.456226 24454 raft_consensus.cc:482] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:16.456339 24454 raft_consensus.cc:3037] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.458367 24454 raft_consensus.cc:504] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.458549 24454 leader_election.cc:302] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:16.458756 24454 leader_election.cc:288] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:16.458804 24405 raft_consensus.cc:2781] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.458961 24405 raft_consensus.cc:686] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:16.459179 24405 consensus_queue.cc:237] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:16.459319 24454 ts_tablet_manager.cc:1398] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240503 14:12:16.461232 23317 catalog_manager.cc:5484] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
W20240503 14:12:16.784337 24464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.784618 24464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:16.791764 24464 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 105.089 ms
time per row: 0.0525444 ms
I20240503 14:12:17.031837 23323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60850:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:17.039925 23577 tablet_service.cc:1467] Processing CreateTablet for tablet feacaf3f7e794a19b20a4dea58730842 (DEFAULT_TABLE table=rebalance_test_table_3 [id=14847a692cab4bf487630962c7ba1934]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.040338 23577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feacaf3f7e794a19b20a4dea58730842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.040444 23447 tablet_service.cc:1467] Processing CreateTablet for tablet ed49476f299e4e8793bdfee31165ec8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=14847a692cab4bf487630962c7ba1934]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.040741 23447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed49476f299e4e8793bdfee31165ec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.042860 23578 tablet_service.cc:1467] Processing CreateTablet for tablet 60aad763f40242f98297320e9e8cf9ad (DEFAULT_TABLE table=rebalance_test_table_3 [id=14847a692cab4bf487630962c7ba1934]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:17.043227 23578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60aad763f40242f98297320e9e8cf9ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:17.045012 24506 tablet_bootstrap.cc:492] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: Bootstrap starting.
I20240503 14:12:17.046420 24506 tablet_bootstrap.cc:654] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.046680 24507 tablet_bootstrap.cc:492] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:17.048620 24507 tablet_bootstrap.cc:654] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.049309 24506 tablet_bootstrap.cc:492] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: No bootstrap required, opened a new log
I20240503 14:12:17.049460 24506 ts_tablet_manager.cc:1367] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:12:17.050109 24506 raft_consensus.cc:348] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.050287 24507 tablet_bootstrap.cc:492] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:17.050293 24506 raft_consensus.cc:374] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.050371 24506 raft_consensus.cc:729] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.050413 24507 ts_tablet_manager.cc:1367] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 14:12:17.050601 24506 consensus_queue.cc:260] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.050804 24506 raft_consensus.cc:388] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:17.050892 24506 raft_consensus.cc:482] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:17.051007 24506 raft_consensus.cc:3037] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.051165 24507 raft_consensus.cc:348] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:17.051331 24507 raft_consensus.cc:374] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.051465 24507 raft_consensus.cc:729] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.051720 24507 consensus_queue.cc:260] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:17.051900 24507 raft_consensus.cc:388] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:17.051995 24507 raft_consensus.cc:482] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:17.052111 24507 raft_consensus.cc:3037] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.053548 24506 raft_consensus.cc:504] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.053861 24506 leader_election.cc:302] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:17.054072 24506 leader_election.cc:288] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:17.054181 24400 raft_consensus.cc:2781] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.054842 24506 ts_tablet_manager.cc:1398] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20240503 14:12:17.054808 24507 raft_consensus.cc:504] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:17.055099 24507 leader_election.cc:302] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:17.055171 24506 tablet_bootstrap.cc:492] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: Bootstrap starting.
I20240503 14:12:17.055444 24459 raft_consensus.cc:2781] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.055583 24459 raft_consensus.cc:686] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:17.055833 24459 consensus_queue.cc:237] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:17.057276 24506 tablet_bootstrap.cc:654] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:17.059695 24506 tablet_bootstrap.cc:492] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: No bootstrap required, opened a new log
I20240503 14:12:17.060034 24506 ts_tablet_manager.cc:1367] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 14:12:17.061044 24506 raft_consensus.cc:348] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.061327 24506 raft_consensus.cc:374] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:17.061444 24506 raft_consensus.cc:729] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Initialized, Role: FOLLOWER
I20240503 14:12:17.061563 24507 leader_election.cc:288] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:17.061717 24506 consensus_queue.cc:260] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.061924 24506 raft_consensus.cc:388] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:17.062070 24506 raft_consensus.cc:482] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:17.062173 24506 raft_consensus.cc:3037] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:17.062320 24507 ts_tablet_manager.cc:1398] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.012s user 0.002s sys 0.004s
I20240503 14:12:17.065706 24506 raft_consensus.cc:504] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.065999 24506 leader_election.cc:302] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:17.066251 24506 leader_election.cc:288] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:17.066339 24510 raft_consensus.cc:2781] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:17.066682 24400 raft_consensus.cc:686] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:17.066756 24506 ts_tablet_manager.cc:1398] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:12:17.066920 24400 consensus_queue.cc:237] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.068967 23323 catalog_manager.cc:5484] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
I20240503 14:12:17.069598 23326 catalog_manager.cc:5484] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } }
I20240503 14:12:17.074934 24510 raft_consensus.cc:686] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:17.075206 24510 consensus_queue.cc:237] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:17.077368 23326 catalog_manager.cc:5484] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } }
W20240503 14:12:17.391264 24518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:17.391575 24518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:17.398768 24518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 118.596 ms
time per row: 0.0592981 ms
-----------------------------------------------------
Suppressions used:
count bytes template
18 4816 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:18.567483 23323 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60850:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:18.575570 23447 tablet_service.cc:1467] Processing CreateTablet for tablet 76c35354470b4d2187c8dd6c0d1594f4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=efb3d2e243c246a393e534a59a399d94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.575721 23578 tablet_service.cc:1467] Processing CreateTablet for tablet 1045468e57c1484995277f9ad5fc20b1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=efb3d2e243c246a393e534a59a399d94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.575942 23446 tablet_service.cc:1467] Processing CreateTablet for tablet 60a2f6744aa741d9b2af7eeefe868184 (DEFAULT_TABLE table=rebalance_test_table_4 [id=efb3d2e243c246a393e534a59a399d94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:18.576045 23447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c35354470b4d2187c8dd6c0d1594f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.576092 23578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1045468e57c1484995277f9ad5fc20b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.576257 23446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a2f6744aa741d9b2af7eeefe868184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:18.580633 24560 tablet_bootstrap.cc:492] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:18.581908 24560 tablet_bootstrap.cc:654] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.582531 24561 tablet_bootstrap.cc:492] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064: Bootstrap starting.
I20240503 14:12:18.583760 24560 tablet_bootstrap.cc:492] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:18.583889 24560 ts_tablet_manager.cc:1367] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:18.584110 24561 tablet_bootstrap.cc:654] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.584667 24560 raft_consensus.cc:348] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.584846 24560 raft_consensus.cc:374] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.584967 24560 raft_consensus.cc:729] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.585211 24560 consensus_queue.cc:260] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.585398 24560 raft_consensus.cc:388] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:18.585484 24560 raft_consensus.cc:482] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:18.585598 24560 raft_consensus.cc:3037] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.585922 24561 tablet_bootstrap.cc:492] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064: No bootstrap required, opened a new log
I20240503 14:12:18.586089 24561 ts_tablet_manager.cc:1367] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:12:18.586823 24561 raft_consensus.cc:348] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:18.586983 24561 raft_consensus.cc:374] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.587074 24561 raft_consensus.cc:729] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.587301 24561 consensus_queue.cc:260] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:18.587473 24561 raft_consensus.cc:388] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:18.587544 24561 raft_consensus.cc:482] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:18.587646 24561 raft_consensus.cc:3037] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.588106 24560 raft_consensus.cc:504] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.588352 24560 leader_election.cc:302] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:18.588757 24560 leader_election.cc:288] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:18.588848 24564 raft_consensus.cc:2781] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.590597 24561 raft_consensus.cc:504] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:18.590701 24564 raft_consensus.cc:686] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:18.590955 24561 leader_election.cc:302] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:18.591068 24564 consensus_queue.cc:237] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.591375 24561 leader_election.cc:288] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:18.591485 24567 raft_consensus.cc:2781] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.595233 24561 ts_tablet_manager.cc:1398] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s
I20240503 14:12:18.596747 24567 raft_consensus.cc:686] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:18.596879 24560 ts_tablet_manager.cc:1398] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20240503 14:12:18.597154 24560 tablet_bootstrap.cc:492] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: Bootstrap starting.
I20240503 14:12:18.597158 24567 consensus_queue.cc:237] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:18.597556 23323 catalog_manager.cc:5484] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
I20240503 14:12:18.598790 24560 tablet_bootstrap.cc:654] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:18.601642 23323 catalog_manager.cc:5484] T 1045468e57c1484995277f9ad5fc20b1 P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } }
I20240503 14:12:18.607230 24560 tablet_bootstrap.cc:492] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: No bootstrap required, opened a new log
I20240503 14:12:18.607388 24560 ts_tablet_manager.cc:1367] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240503 14:12:18.608026 24560 raft_consensus.cc:348] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.608253 24560 raft_consensus.cc:374] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:18.608341 24560 raft_consensus.cc:729] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Initialized, Role: FOLLOWER
I20240503 14:12:18.608580 24560 consensus_queue.cc:260] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.608762 24560 raft_consensus.cc:388] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:18.608867 24560 raft_consensus.cc:482] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:18.609004 24560 raft_consensus.cc:3037] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:18.610844 24560 raft_consensus.cc:504] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.611018 24560 leader_election.cc:302] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:18.611232 24560 leader_election.cc:288] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:18.611275 24564 raft_consensus.cc:2781] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:18.611661 24564 raft_consensus.cc:686] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:18.611687 24560 ts_tablet_manager.cc:1398] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:12:18.611932 24564 consensus_queue.cc:237] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:18.613773 23324 catalog_manager.cc:5484] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } }
W20240503 14:12:18.931946 24576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:18.932251 24576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:18.939604 24576 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 109.666 ms
time per row: 0.0548332 ms
I20240503 14:12:19.187530 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:41087
--local_ip_for_outbound_sockets=127.20.240.67
--tserver_master_addrs=127.20.240.126:43899
--webserver_port=44429
--webserver_interface=127.20.240.67
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:12:19.491564 24615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:19.491816 24615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:19.491935 24615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:19.499056 24615 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:19.499186 24615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:19.499354 24615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20240503 14:12:19.507668 24615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:41087
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=44429
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:19.508857 24615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:19.510542 24615 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:19.522701 24620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:19.523535 24621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:19.524204 24623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:19.524911 24615 server_base.cc:1034] running on GCE node
I20240503 14:12:19.525408 24615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:19.526212 24615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:19.527455 24615 hybrid_clock.cc:648] HybridClock initialized: now 1714745539527418 us; error 36 us; skew 500 ppm
I20240503 14:12:19.527772 24615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:19.529847 24615 webserver.cc:460] Webserver started at http://127.20.240.67:44429/ using document root <none> and password file <none>
I20240503 14:12:19.530611 24615 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:19.530758 24615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:19.535601 24615 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240503 14:12:19.538069 24629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:19.539305 24615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:12:19.539444 24615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1a4466679fbc43618c8fb2e3d8a26dfd"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-xhmt"
I20240503 14:12:19.540194 24615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:19.553824 24615 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:19.554533 24615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:19.554803 24615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:19.555655 24615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:19.557097 24615 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:19.557170 24615 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:19.557312 24615 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:19.557363 24615 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:19.591414 24615 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:41087
I20240503 14:12:19.591460 24741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:41087 every 8 connection(s)
I20240503 14:12:19.592757 24615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:19.602325 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 24615
I20240503 14:12:19.602934 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.68:44039
--local_ip_for_outbound_sockets=127.20.240.68
--tserver_master_addrs=127.20.240.126:43899
--webserver_port=42509
--webserver_interface=127.20.240.68
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:19.604807 24742 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:19.605087 24742 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:19.605727 24742 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:19.607116 23326 ts_manager.cc:194] Re-registered known tserver with Master: 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67:41087)
I20240503 14:12:19.607868 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:39147
W20240503 14:12:19.916749 24746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:19.917014 24746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:19.917140 24746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:19.924115 24746 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:19.924216 24746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:19.924345 24746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.68
I20240503 14:12:19.932636 24746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.68:44039
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.240.68
--webserver_port=42509
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.68
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:19.933921 24746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:19.935568 24746 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:19.947523 24751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:19.948024 24752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:19.948593 24754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:19.949638 24746 server_base.cc:1034] running on GCE node
I20240503 14:12:19.950192 24746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:19.951063 24746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:19.952250 24746 hybrid_clock.cc:648] HybridClock initialized: now 1714745539952207 us; error 42 us; skew 500 ppm
I20240503 14:12:19.952535 24746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:19.961632 24746 webserver.cc:460] Webserver started at http://127.20.240.68:42509/ using document root <none> and password file <none>
I20240503 14:12:19.962390 24746 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:19.962515 24746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:19.967317 24746 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:19.969882 24760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:19.971068 24746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:19.971221 24746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2062470ed9164d389aa2e60973ecdcdb"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:19.972030 24746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:19.991819 24746 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:19.992550 24746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:19.992815 24746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:19.993662 24746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:19.995370 24767 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 14:12:20.000216 24746 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 14:12:20.000329 24746 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20240503 14:12:20.000461 24746 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 14:12:20.003401 24746 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 14:12:20.003497 24746 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:20.003791 24767 tablet_bootstrap.cc:492] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Bootstrap starting.
I20240503 14:12:20.023955 24767 log.cc:824] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:20.029278 24767 tablet_bootstrap.cc:492] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:20.030058 24767 tablet_bootstrap.cc:492] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Bootstrap complete.
I20240503 14:12:20.031533 24767 ts_tablet_manager.cc:1367] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.009s
I20240503 14:12:20.038574 24746 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.68:44039
I20240503 14:12:20.038679 24874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.68:44039 every 8 connection(s)
I20240503 14:12:20.040007 24746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:20.042522 24767 raft_consensus.cc:348] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:20.043397 24767 raft_consensus.cc:729] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2062470ed9164d389aa2e60973ecdcdb, State: Initialized, Role: FOLLOWER
I20240503 14:12:20.044083 24767 consensus_queue.cc:260] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:20.044302 24767 raft_consensus.cc:388] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:20.044469 24767 raft_consensus.cc:482] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:20.044636 24767 raft_consensus.cc:3037] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:20.048879 24767 raft_consensus.cc:504] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:20.049708 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 24746
I20240503 14:12:20.049432 24767 leader_election.cc:302] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2062470ed9164d389aa2e60973ecdcdb; no voters:
I20240503 14:12:20.050424 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:38795
--local_ip_for_outbound_sockets=127.20.240.69
--tserver_master_addrs=127.20.240.126:43899
--webserver_port=35873
--webserver_interface=127.20.240.69
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:20.050473 24767 leader_election.cc:288] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 14:12:20.050596 24878 raft_consensus.cc:2781] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:20.054747 24878 raft_consensus.cc:686] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [term 2 LEADER]: Becoming Leader. State: Replica: 2062470ed9164d389aa2e60973ecdcdb, State: Running, Role: LEADER
I20240503 14:12:20.055342 24878 consensus_queue.cc:237] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:20.055863 24875 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:20.055917 24767 ts_tablet_manager.cc:1398] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb: Time spent starting tablet: real 0.024s user 0.020s sys 0.003s
I20240503 14:12:20.056185 24875 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:20.057186 24875 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:20.059748 23326 ts_manager.cc:194] Re-registered known tserver with Master: 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68:44039)
I20240503 14:12:20.060148 23326 catalog_manager.cc:5484] T db65a7aa1497429cb05bcd6b94d88914 P 2062470ed9164d389aa2e60973ecdcdb reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "2062470ed9164d389aa2e60973ecdcdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:20.064893 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.68:47283
I20240503 14:12:20.068045 24875 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
W20240503 14:12:20.371690 24881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:20.371975 24881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:20.372076 24881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:20.379776 24881 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:20.379889 24881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:20.379995 24881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:12:20.388725 24881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:38795
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=35873
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:20.389919 24881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:20.391534 24881 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:20.403656 24892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:20.404260 24895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:20.405154 24893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:20.405970 24881 server_base.cc:1034] running on GCE node
I20240503 14:12:20.406595 24881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:20.407444 24881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:20.408632 24881 hybrid_clock.cc:648] HybridClock initialized: now 1714745540408576 us; error 47 us; skew 500 ppm
I20240503 14:12:20.408900 24881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:20.410971 24881 webserver.cc:460] Webserver started at http://127.20.240.69:35873/ using document root <none> and password file <none>
I20240503 14:12:20.411612 24881 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:20.411723 24881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:20.416388 24881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:20.418838 24901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:20.420009 24881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:20.420131 24881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4fbbe1e5ec864182b3116ef1278f7a3a"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-xhmt"
I20240503 14:12:20.420872 24881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:20.445281 24881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:20.445950 24881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:20.446182 24881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:20.447026 24881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:20.448359 24881 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:20.448427 24881 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:20.448513 24881 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:20.448555 24881 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:20.478498 24881 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:38795
I20240503 14:12:20.478542 25013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:38795 every 8 connection(s)
I20240503 14:12:20.479818 24881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:20.489925 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 24881
I20240503 14:12:20.490594 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.70:34091
--local_ip_for_outbound_sockets=127.20.240.70
--tserver_master_addrs=127.20.240.126:43899
--webserver_port=35929
--webserver_interface=127.20.240.70
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:20.491127 25014 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:20.491381 25014 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:20.492028 25014 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:20.493283 23326 ts_manager.cc:194] Re-registered known tserver with Master: 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69:38795)
I20240503 14:12:20.494073 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:44395
I20240503 14:12:20.610303 24742 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
W20240503 14:12:20.800340 25017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:20.800594 25017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:20.800729 25017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:20.808480 25017 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:20.808583 25017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:20.808729 25017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.70
I20240503 14:12:20.817250 25017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.70:34091
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.240.70
--webserver_port=35929
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.70
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:20.818456 25017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:20.820040 25017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:20.832430 25022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:20.833338 25023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:20.834023 25025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:20.834738 25017 server_base.cc:1034] running on GCE node
I20240503 14:12:20.835300 25017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:20.836243 25017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:20.837435 25017 hybrid_clock.cc:648] HybridClock initialized: now 1714745540837385 us; error 44 us; skew 500 ppm
I20240503 14:12:20.837705 25017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:20.839757 25017 webserver.cc:460] Webserver started at http://127.20.240.70:35929/ using document root <none> and password file <none>
I20240503 14:12:20.840392 25017 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:20.840497 25017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:20.845327 25017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:20.847868 25031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:20.849061 25017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:20.849212 25017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "95dfe9d1f79f4a8696fc5586e3ea3731"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:20.849962 25017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:20.872757 25017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:20.873564 25017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:20.873884 25017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:20.874775 25017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:20.876257 25017 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:20.876327 25017 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:20.876415 25017 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:20.876457 25017 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:20.906044 25017 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.70:34091
I20240503 14:12:20.906062 25143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.70:34091 every 8 connection(s)
I20240503 14:12:20.907423 25017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:20.916404 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 25017
I20240503 14:12:20.917107 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.71:43179
--local_ip_for_outbound_sockets=127.20.240.71
--tserver_master_addrs=127.20.240.126:43899
--webserver_port=38509
--webserver_interface=127.20.240.71
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:20.919641 25144 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:20.919899 25144 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:20.920529 25144 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:20.921691 23326 ts_manager.cc:194] Re-registered known tserver with Master: 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70:34091)
I20240503 14:12:20.922498 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.70:40993
W20240503 14:12:21.241706 25147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:21.241993 25147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:21.242070 25147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:21.249466 25147 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:21.249568 25147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:21.249696 25147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.71
I20240503 14:12:21.259045 25147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.71:43179
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.240.71
--webserver_port=38509
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.71
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:21.260268 25147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:21.261942 25147 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:21.274173 25155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:21.274778 25153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:21.275710 25152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:21.276574 25147 server_base.cc:1034] running on GCE node
I20240503 14:12:21.277129 25147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:21.277961 25147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:21.279151 25147 hybrid_clock.cc:648] HybridClock initialized: now 1714745541279107 us; error 38 us; skew 500 ppm
I20240503 14:12:21.279439 25147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:21.281863 25147 webserver.cc:460] Webserver started at http://127.20.240.71:38509/ using document root <none> and password file <none>
I20240503 14:12:21.282572 25147 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:21.282706 25147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:21.287283 25147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:21.289707 25161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:21.290916 25147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:12:21.291064 25147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "0e71aaeb321d4364bc1d81d64815b201"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-xhmt"
I20240503 14:12:21.291827 25147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:21.306037 25147 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:21.306713 25147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:21.306962 25147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:21.307825 25147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:21.309187 25147 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:21.309262 25147 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:21.309383 25147 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:21.309435 25147 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:21.342499 25147 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.71:43179
I20240503 14:12:21.342530 25273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.71:43179 every 8 connection(s)
I20240503 14:12:21.343967 25147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:21.354219 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 25147
I20240503 14:12:21.356606 25274 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:21.356984 25274 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:21.357915 25274 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:21.359367 23326 ts_manager.cc:194] Re-registered known tserver with Master: 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71:43179)
I20240503 14:12:21.360229 23326 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.71:38803
I20240503 14:12:21.496321 25014 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
I20240503 14:12:21.812505 25209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:21.833158 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:21.870733 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:21.872546 24677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:21.877729 23578 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:21.886842 23447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:21.914336 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:21.924572 25144 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
I20240503 14:12:22.082231 25209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:22.089437 24677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:22.098774 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:22.099459 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:22.137593 23447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:22.146644 23578 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:22.154369 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:22.232813 23598 consensus_queue.cc:237] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
W20240503 14:12:22.241145 23532 consensus_peers.cc:485] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 -> Peer 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70:34091): Couldn't send request to peer 95dfe9d1f79f4a8696fc5586e3ea3731. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: feacaf3f7e794a19b20a4dea58730842. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:22.362684 25274 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
I20240503 14:12:22.806623 25375 ts_tablet_manager.cc:902] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: Initiating tablet copy from peer 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:22.808322 25375 tablet_copy_client.cc:308] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:36169
I20240503 14:12:22.815115 23618 tablet_copy_service.cc:140] P 7f3556100c8d46aaafb47e87defe8064: Received BeginTabletCopySession request for tablet feacaf3f7e794a19b20a4dea58730842 from peer 95dfe9d1f79f4a8696fc5586e3ea3731 ({username='slave'} at 127.20.240.70:34297)
I20240503 14:12:22.815317 23618 tablet_copy_service.cc:161] P 7f3556100c8d46aaafb47e87defe8064: Beginning new tablet copy session on tablet feacaf3f7e794a19b20a4dea58730842 from peer 95dfe9d1f79f4a8696fc5586e3ea3731 at {username='slave'} at 127.20.240.70:34297: session id = 95dfe9d1f79f4a8696fc5586e3ea3731-feacaf3f7e794a19b20a4dea58730842
I20240503 14:12:22.817914 23618 tablet_copy_source_session.cc:205] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:22.822448 25375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feacaf3f7e794a19b20a4dea58730842. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.831846 25375 tablet_copy_client.cc:783] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:22.832118 25375 tablet_copy_client.cc:647] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:22.834424 25375 tablet_copy_client.cc:523] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:22.837704 25375 tablet_bootstrap.cc:492] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: Bootstrap starting.
I20240503 14:12:22.846114 25375 log.cc:824] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:22.877638 25375 tablet_bootstrap.cc:492] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:22.878155 25375 tablet_bootstrap.cc:492] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: Bootstrap complete.
I20240503 14:12:22.878556 25375 ts_tablet_manager.cc:1367] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: Time spent bootstrapping tablet: real 0.041s user 0.039s sys 0.004s
I20240503 14:12:22.882809 25375 raft_consensus.cc:348] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:22.883610 25375 raft_consensus.cc:729] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95dfe9d1f79f4a8696fc5586e3ea3731, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.884049 25375 consensus_queue.cc:260] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:22.885829 25375 ts_tablet_manager.cc:1398] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240503 14:12:22.886880 23618 tablet_copy_service.cc:342] P 7f3556100c8d46aaafb47e87defe8064: Request end of tablet copy session 95dfe9d1f79f4a8696fc5586e3ea3731-feacaf3f7e794a19b20a4dea58730842 received from {username='slave'} at 127.20.240.70:34297
I20240503 14:12:22.887063 23618 tablet_copy_service.cc:433] P 7f3556100c8d46aaafb47e87defe8064: ending tablet copy session 95dfe9d1f79f4a8696fc5586e3ea3731-feacaf3f7e794a19b20a4dea58730842 on tablet feacaf3f7e794a19b20a4dea58730842 with peer 95dfe9d1f79f4a8696fc5586e3ea3731
I20240503 14:12:23.364349 25099 raft_consensus.cc:1203] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:23.366339 25385 raft_consensus.cc:2932] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:23.367187 25099 raft_consensus.cc:2932] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:23.369470 25385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52700 (request call id 0) took 1137 ms. Trace:
I20240503 14:12:23.369709 25385 rpcz_store.cc:276] 0503 14:12:22.231784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:22.231838 (+ 54us) service_pool.cc:225] Handling call
0503 14:12:22.233963 (+ 2125us) log.cc:842] Serialized 260 byte log entry
0503 14:12:23.369418 (+1135455us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":61,"raft.queue_time_us":383,"raft.run_cpu_time_us":524,"raft.run_wall_time_us":523,"spinlock_wait_cycles":6016,"thread_start_us":432,"threads_started":2,"wal-append.queue_time_us":307,"wal-append.run_cpu_time_us":629,"wal-append.run_wall_time_us":1000684}
I20240503 14:12:23.373746 23325 catalog_manager.cc:5484] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 reported cstate change: config changed from index -1 to 4, VOTER 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:23.406674 23467 consensus_queue.cc:237] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
W20240503 14:12:23.415457 23402 consensus_peers.cc:485] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 -> Peer 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67:41087): Couldn't send request to peer 1a4466679fbc43618c8fb2e3d8a26dfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d433f9489b204c8f9e1ef5c2164c5a6e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:23.872516 25413 ts_tablet_manager.cc:902] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: Initiating tablet copy from peer 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:23.874344 25413 tablet_copy_client.cc:308] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:36725
I20240503 14:12:23.880479 23487 tablet_copy_service.cc:140] P 5ee9b8f29d3349f4ba05425d5287ff58: Received BeginTabletCopySession request for tablet d433f9489b204c8f9e1ef5c2164c5a6e from peer 1a4466679fbc43618c8fb2e3d8a26dfd ({username='slave'} at 127.20.240.67:51827)
I20240503 14:12:23.880709 23487 tablet_copy_service.cc:161] P 5ee9b8f29d3349f4ba05425d5287ff58: Beginning new tablet copy session on tablet d433f9489b204c8f9e1ef5c2164c5a6e from peer 1a4466679fbc43618c8fb2e3d8a26dfd at {username='slave'} at 127.20.240.67:51827: session id = 1a4466679fbc43618c8fb2e3d8a26dfd-d433f9489b204c8f9e1ef5c2164c5a6e
I20240503 14:12:23.883787 23487 tablet_copy_source_session.cc:205] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:23.888047 25413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d433f9489b204c8f9e1ef5c2164c5a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:23.897346 25413 tablet_copy_client.cc:783] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:23.897631 25413 tablet_copy_client.cc:647] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:23.899964 25413 tablet_copy_client.cc:523] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:23.902784 25413 tablet_bootstrap.cc:492] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: Bootstrap starting.
I20240503 14:12:23.911857 25413 log.cc:824] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:23.941679 25413 tablet_bootstrap.cc:492] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:23.942209 25413 tablet_bootstrap.cc:492] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: Bootstrap complete.
I20240503 14:12:23.942624 25413 ts_tablet_manager.cc:1367] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.004s
I20240503 14:12:23.946801 25413 raft_consensus.cc:348] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:23.947556 25413 raft_consensus.cc:729] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4466679fbc43618c8fb2e3d8a26dfd, State: Initialized, Role: FOLLOWER
I20240503 14:12:23.947952 25413 consensus_queue.cc:260] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:23.949381 25413 ts_tablet_manager.cc:1398] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:12:23.950372 23487 tablet_copy_service.cc:342] P 5ee9b8f29d3349f4ba05425d5287ff58: Request end of tablet copy session 1a4466679fbc43618c8fb2e3d8a26dfd-d433f9489b204c8f9e1ef5c2164c5a6e received from {username='slave'} at 127.20.240.67:51827
I20240503 14:12:23.950536 23487 tablet_copy_service.cc:433] P 5ee9b8f29d3349f4ba05425d5287ff58: ending tablet copy session 1a4466679fbc43618c8fb2e3d8a26dfd-d433f9489b204c8f9e1ef5c2164c5a6e on tablet d433f9489b204c8f9e1ef5c2164c5a6e with peer 1a4466679fbc43618c8fb2e3d8a26dfd
I20240503 14:12:24.417447 24697 raft_consensus.cc:1203] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:24.419399 25422 raft_consensus.cc:2932] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:24.420238 24697 raft_consensus.cc:2932] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:24.423563 25422 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:45988 (request call id 0) took 1017 ms. Trace:
I20240503 14:12:24.423770 25422 rpcz_store.cc:276] 0503 14:12:23.405618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:23.405689 (+ 71us) service_pool.cc:225] Handling call
0503 14:12:23.407814 (+ 2125us) log.cc:842] Serialized 260 byte log entry
0503 14:12:24.423507 (+1015693us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":61,"raft.queue_time_us":306,"raft.run_cpu_time_us":529,"raft.run_wall_time_us":529,"spinlock_wait_cycles":75008,"thread_start_us":414,"threads_started":2,"wal-append.queue_time_us":283,"wal-append.run_cpu_time_us":616,"wal-append.run_wall_time_us":1000695}
I20240503 14:12:24.425604 23321 catalog_manager.cc:5484] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd reported cstate change: config changed from index -1 to 4, VOTER 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:24.454645 23598 consensus_queue.cc:237] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
W20240503 14:12:24.461935 23532 consensus_peers.cc:485] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 -> Peer 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69:38795): Couldn't send request to peer 4fbbe1e5ec864182b3116ef1278f7a3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54ff6aaa60734f548ff2b8da41ea5b3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:25.077977 25450 ts_tablet_manager.cc:902] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: Initiating tablet copy from peer 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:25.079919 25450 tablet_copy_client.cc:308] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:36169
I20240503 14:12:25.086277 23618 tablet_copy_service.cc:140] P 7f3556100c8d46aaafb47e87defe8064: Received BeginTabletCopySession request for tablet 54ff6aaa60734f548ff2b8da41ea5b3a from peer 4fbbe1e5ec864182b3116ef1278f7a3a ({username='slave'} at 127.20.240.69:60427)
I20240503 14:12:25.086436 23618 tablet_copy_service.cc:161] P 7f3556100c8d46aaafb47e87defe8064: Beginning new tablet copy session on tablet 54ff6aaa60734f548ff2b8da41ea5b3a from peer 4fbbe1e5ec864182b3116ef1278f7a3a at {username='slave'} at 127.20.240.69:60427: session id = 4fbbe1e5ec864182b3116ef1278f7a3a-54ff6aaa60734f548ff2b8da41ea5b3a
I20240503 14:12:25.088724 23618 tablet_copy_source_session.cc:205] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:25.092727 25450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54ff6aaa60734f548ff2b8da41ea5b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.102072 25450 tablet_copy_client.cc:783] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:25.102373 25450 tablet_copy_client.cc:647] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:25.104223 25450 tablet_copy_client.cc:523] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:25.106953 25450 tablet_bootstrap.cc:492] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: Bootstrap starting.
I20240503 14:12:25.115788 25450 log.cc:824] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:25.146075 25450 tablet_bootstrap.cc:492] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:25.146766 25450 tablet_bootstrap.cc:492] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: Bootstrap complete.
I20240503 14:12:25.147171 25450 ts_tablet_manager.cc:1367] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: Time spent bootstrapping tablet: real 0.040s user 0.036s sys 0.004s
I20240503 14:12:25.151393 25450 raft_consensus.cc:348] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:25.152140 25450 raft_consensus.cc:729] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fbbe1e5ec864182b3116ef1278f7a3a, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.152508 25450 consensus_queue.cc:260] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:25.153873 25450 ts_tablet_manager.cc:1398] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20240503 14:12:25.154850 23618 tablet_copy_service.cc:342] P 7f3556100c8d46aaafb47e87defe8064: Request end of tablet copy session 4fbbe1e5ec864182b3116ef1278f7a3a-54ff6aaa60734f548ff2b8da41ea5b3a received from {username='slave'} at 127.20.240.69:60427
I20240503 14:12:25.154973 23618 tablet_copy_service.cc:433] P 7f3556100c8d46aaafb47e87defe8064: ending tablet copy session 4fbbe1e5ec864182b3116ef1278f7a3a-54ff6aaa60734f548ff2b8da41ea5b3a on tablet 54ff6aaa60734f548ff2b8da41ea5b3a with peer 4fbbe1e5ec864182b3116ef1278f7a3a
I20240503 14:12:25.700403 24969 raft_consensus.cc:1203] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:25.702330 25461 raft_consensus.cc:2932] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:25.703125 24969 raft_consensus.cc:2932] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:25.705106 25461 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52714 (request call id 0) took 1251 ms. Trace:
I20240503 14:12:25.705312 25461 rpcz_store.cc:276] 0503 14:12:24.454048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:24.454106 (+ 58us) service_pool.cc:225] Handling call
0503 14:12:24.455286 (+ 1180us) log.cc:842] Serialized 260 byte log entry
0503 14:12:25.705058 (+1249772us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":64,"raft.queue_time_us":55,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":219,"thread_start_us":170,"threads_started":1,"wal-append.queue_time_us":246,"wal-append.run_cpu_time_us":502,"wal-append.run_wall_time_us":1000574}
I20240503 14:12:25.707152 23321 catalog_manager.cc:5484] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: config changed from index -1 to 4, VOTER 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:25.741448 23467 consensus_queue.cc:237] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
W20240503 14:12:25.749346 23402 consensus_peers.cc:485] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 -> Peer 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71:43179): Couldn't send request to peer 0e71aaeb321d4364bc1d81d64815b201. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76c35354470b4d2187c8dd6c0d1594f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:26.220161 25488 ts_tablet_manager.cc:902] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: Initiating tablet copy from peer 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:26.221735 25488 tablet_copy_client.cc:308] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:36725
I20240503 14:12:26.228116 23487 tablet_copy_service.cc:140] P 5ee9b8f29d3349f4ba05425d5287ff58: Received BeginTabletCopySession request for tablet 76c35354470b4d2187c8dd6c0d1594f4 from peer 0e71aaeb321d4364bc1d81d64815b201 ({username='slave'} at 127.20.240.71:47123)
I20240503 14:12:26.228250 23487 tablet_copy_service.cc:161] P 5ee9b8f29d3349f4ba05425d5287ff58: Beginning new tablet copy session on tablet 76c35354470b4d2187c8dd6c0d1594f4 from peer 0e71aaeb321d4364bc1d81d64815b201 at {username='slave'} at 127.20.240.71:47123: session id = 0e71aaeb321d4364bc1d81d64815b201-76c35354470b4d2187c8dd6c0d1594f4
I20240503 14:12:26.230072 23487 tablet_copy_source_session.cc:205] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:26.234269 25488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c35354470b4d2187c8dd6c0d1594f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:26.244072 25488 tablet_copy_client.cc:783] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:26.244344 25488 tablet_copy_client.cc:647] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:26.246269 25488 tablet_copy_client.cc:523] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:26.249240 25488 tablet_bootstrap.cc:492] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: Bootstrap starting.
I20240503 14:12:26.257647 25488 log.cc:824] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:26.290282 25488 tablet_bootstrap.cc:492] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:26.290854 25488 tablet_bootstrap.cc:492] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: Bootstrap complete.
I20240503 14:12:26.291254 25488 ts_tablet_manager.cc:1367] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: Time spent bootstrapping tablet: real 0.042s user 0.040s sys 0.004s
I20240503 14:12:26.295542 25488 raft_consensus.cc:348] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:26.296340 25488 raft_consensus.cc:729] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e71aaeb321d4364bc1d81d64815b201, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.296747 25488 consensus_queue.cc:260] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:26.298321 25488 ts_tablet_manager.cc:1398] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20240503 14:12:26.299338 23487 tablet_copy_service.cc:342] P 5ee9b8f29d3349f4ba05425d5287ff58: Request end of tablet copy session 0e71aaeb321d4364bc1d81d64815b201-76c35354470b4d2187c8dd6c0d1594f4 received from {username='slave'} at 127.20.240.71:47123
I20240503 14:12:26.299479 23487 tablet_copy_service.cc:433] P 5ee9b8f29d3349f4ba05425d5287ff58: ending tablet copy session 0e71aaeb321d4364bc1d81d64815b201-76c35354470b4d2187c8dd6c0d1594f4 on tablet 76c35354470b4d2187c8dd6c0d1594f4 with peer 0e71aaeb321d4364bc1d81d64815b201
I20240503 14:12:26.637882 25229 raft_consensus.cc:1203] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:26.639592 25496 raft_consensus.cc:2932] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:26.640156 25229 raft_consensus.cc:2932] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:26.645802 23321 catalog_manager.cc:5484] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 reported cstate change: config changed from index -1 to 4, VOTER 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:26.673074 23467 consensus_queue.cc:237] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
W20240503 14:12:26.681064 23401 consensus_peers.cc:485] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 -> Peer 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68:44039): Couldn't send request to peer 2062470ed9164d389aa2e60973ecdcdb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60a2f6744aa741d9b2af7eeefe868184. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:27.296311 25520 ts_tablet_manager.cc:902] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: Initiating tablet copy from peer 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:27.297938 25520 tablet_copy_client.cc:308] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:36725
I20240503 14:12:27.303721 23487 tablet_copy_service.cc:140] P 5ee9b8f29d3349f4ba05425d5287ff58: Received BeginTabletCopySession request for tablet 60a2f6744aa741d9b2af7eeefe868184 from peer 2062470ed9164d389aa2e60973ecdcdb ({username='slave'} at 127.20.240.68:38589)
I20240503 14:12:27.303880 23487 tablet_copy_service.cc:161] P 5ee9b8f29d3349f4ba05425d5287ff58: Beginning new tablet copy session on tablet 60a2f6744aa741d9b2af7eeefe868184 from peer 2062470ed9164d389aa2e60973ecdcdb at {username='slave'} at 127.20.240.68:38589: session id = 2062470ed9164d389aa2e60973ecdcdb-60a2f6744aa741d9b2af7eeefe868184
I20240503 14:12:27.305673 23487 tablet_copy_source_session.cc:205] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:27.307731 25520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60a2f6744aa741d9b2af7eeefe868184. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:27.312976 25520 tablet_copy_client.cc:783] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:27.313205 25520 tablet_copy_client.cc:647] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:27.315023 25520 tablet_copy_client.cc:523] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:27.317884 25520 tablet_bootstrap.cc:492] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: Bootstrap starting.
I20240503 14:12:27.355567 25520 tablet_bootstrap.cc:492] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:27.355988 25520 tablet_bootstrap.cc:492] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: Bootstrap complete.
I20240503 14:12:27.356215 25520 ts_tablet_manager.cc:1367] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.012s
I20240503 14:12:27.356964 25520 raft_consensus.cc:348] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:27.357362 25520 raft_consensus.cc:729] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2062470ed9164d389aa2e60973ecdcdb, State: Initialized, Role: FOLLOWER
I20240503 14:12:27.357523 25520 consensus_queue.cc:260] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:27.358722 25520 ts_tablet_manager.cc:1398] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:12:27.359892 23487 tablet_copy_service.cc:342] P 5ee9b8f29d3349f4ba05425d5287ff58: Request end of tablet copy session 2062470ed9164d389aa2e60973ecdcdb-60a2f6744aa741d9b2af7eeefe868184 received from {username='slave'} at 127.20.240.68:38589
I20240503 14:12:27.360054 23487 tablet_copy_service.cc:433] P 5ee9b8f29d3349f4ba05425d5287ff58: ending tablet copy session 2062470ed9164d389aa2e60973ecdcdb-60a2f6744aa741d9b2af7eeefe868184 on tablet 60a2f6744aa741d9b2af7eeefe868184 with peer 2062470ed9164d389aa2e60973ecdcdb
I20240503 14:12:27.709151 24829 raft_consensus.cc:1203] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:27.710647 25529 raft_consensus.cc:2932] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:27.711294 24829 raft_consensus.cc:2932] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:27.715454 25529 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46018 (request call id 0) took 1042 ms. Trace:
I20240503 14:12:27.715649 25529 rpcz_store.cc:276] 0503 14:12:26.672559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:26.672626 (+ 67us) service_pool.cc:225] Handling call
0503 14:12:26.673663 (+ 1037us) log.cc:842] Serialized 260 byte log entry
0503 14:12:27.715403 (+1041740us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":54,"raft.queue_time_us":48,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":276,"spinlock_wait_cycles":9600,"thread_start_us":213,"threads_started":1,"wal-append.queue_time_us":322,"wal-append.run_cpu_time_us":540,"wal-append.run_wall_time_us":1000597}
I20240503 14:12:27.716957 23321 catalog_manager.cc:5484] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb reported cstate change: config changed from index -1 to 4, VOTER 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:27.744550 23598 consensus_queue.cc:237] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
W20240503 14:12:27.746780 23532 consensus_peers.cc:485] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 -> Peer 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70:34091): Couldn't send request to peer 95dfe9d1f79f4a8696fc5586e3ea3731. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a71476f51de74a25ba84fc92ef0f9414. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:28.169315 25554 ts_tablet_manager.cc:902] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: Initiating tablet copy from peer 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:28.170508 25554 tablet_copy_client.cc:308] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:36169
I20240503 14:12:28.171188 23618 tablet_copy_service.cc:140] P 7f3556100c8d46aaafb47e87defe8064: Received BeginTabletCopySession request for tablet a71476f51de74a25ba84fc92ef0f9414 from peer 95dfe9d1f79f4a8696fc5586e3ea3731 ({username='slave'} at 127.20.240.70:34297)
I20240503 14:12:28.171356 23618 tablet_copy_service.cc:161] P 7f3556100c8d46aaafb47e87defe8064: Beginning new tablet copy session on tablet a71476f51de74a25ba84fc92ef0f9414 from peer 95dfe9d1f79f4a8696fc5586e3ea3731 at {username='slave'} at 127.20.240.70:34297: session id = 95dfe9d1f79f4a8696fc5586e3ea3731-a71476f51de74a25ba84fc92ef0f9414
I20240503 14:12:28.174983 23618 tablet_copy_source_session.cc:205] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:28.176201 25554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a71476f51de74a25ba84fc92ef0f9414. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:28.180423 25554 tablet_copy_client.cc:783] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:28.180682 25554 tablet_copy_client.cc:647] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:28.182314 25554 tablet_copy_client.cc:523] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:28.184489 25554 tablet_bootstrap.cc:492] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: Bootstrap starting.
I20240503 14:12:28.211385 25554 tablet_bootstrap.cc:492] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:28.211762 25554 tablet_bootstrap.cc:492] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: Bootstrap complete.
I20240503 14:12:28.211997 25554 ts_tablet_manager.cc:1367] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: Time spent bootstrapping tablet: real 0.028s user 0.023s sys 0.005s
I20240503 14:12:28.212733 25554 raft_consensus.cc:348] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:28.213101 25554 raft_consensus.cc:729] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95dfe9d1f79f4a8696fc5586e3ea3731, State: Initialized, Role: FOLLOWER
I20240503 14:12:28.213310 25554 consensus_queue.cc:260] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:28.214699 25554 ts_tablet_manager.cc:1398] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:28.215934 23618 tablet_copy_service.cc:342] P 7f3556100c8d46aaafb47e87defe8064: Request end of tablet copy session 95dfe9d1f79f4a8696fc5586e3ea3731-a71476f51de74a25ba84fc92ef0f9414 received from {username='slave'} at 127.20.240.70:34297
I20240503 14:12:28.216068 23618 tablet_copy_service.cc:433] P 7f3556100c8d46aaafb47e87defe8064: ending tablet copy session 95dfe9d1f79f4a8696fc5586e3ea3731-a71476f51de74a25ba84fc92ef0f9414 on tablet a71476f51de74a25ba84fc92ef0f9414 with peer 95dfe9d1f79f4a8696fc5586e3ea3731
I20240503 14:12:28.718210 25099 raft_consensus.cc:1203] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:28.719879 25562 raft_consensus.cc:2932] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:28.720643 25099 raft_consensus.cc:2932] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:28.727752 23326 catalog_manager.cc:5484] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 reported cstate change: config changed from index -1 to 4, VOTER 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:28.758698 23467 consensus_queue.cc:237] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
W20240503 14:12:28.760567 23402 consensus_peers.cc:485] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 -> Peer 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67:41087): Couldn't send request to peer 1a4466679fbc43618c8fb2e3d8a26dfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3aba18dc398d4c8eb1dfce74850a1478. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.203953 25586 ts_tablet_manager.cc:902] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: Initiating tablet copy from peer 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:29.205520 25586 tablet_copy_client.cc:308] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:36725
I20240503 14:12:29.206178 23487 tablet_copy_service.cc:140] P 5ee9b8f29d3349f4ba05425d5287ff58: Received BeginTabletCopySession request for tablet 3aba18dc398d4c8eb1dfce74850a1478 from peer 1a4466679fbc43618c8fb2e3d8a26dfd ({username='slave'} at 127.20.240.67:51827)
I20240503 14:12:29.206377 23487 tablet_copy_service.cc:161] P 5ee9b8f29d3349f4ba05425d5287ff58: Beginning new tablet copy session on tablet 3aba18dc398d4c8eb1dfce74850a1478 from peer 1a4466679fbc43618c8fb2e3d8a26dfd at {username='slave'} at 127.20.240.67:51827: session id = 1a4466679fbc43618c8fb2e3d8a26dfd-3aba18dc398d4c8eb1dfce74850a1478
I20240503 14:12:29.208283 23487 tablet_copy_source_session.cc:205] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.209378 25586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aba18dc398d4c8eb1dfce74850a1478. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.213459 25586 tablet_copy_client.cc:783] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.213683 25586 tablet_copy_client.cc:647] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.215088 25586 tablet_copy_client.cc:523] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.217682 25586 tablet_bootstrap.cc:492] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: Bootstrap starting.
I20240503 14:12:29.246466 25586 tablet_bootstrap.cc:492] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:29.246799 25586 tablet_bootstrap.cc:492] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: Bootstrap complete.
I20240503 14:12:29.247026 25586 ts_tablet_manager.cc:1367] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.004s
I20240503 14:12:29.247826 25586 raft_consensus.cc:348] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:29.248230 25586 raft_consensus.cc:729] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4466679fbc43618c8fb2e3d8a26dfd, State: Initialized, Role: FOLLOWER
I20240503 14:12:29.248409 25586 consensus_queue.cc:260] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:29.249058 25586 ts_tablet_manager.cc:1398] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:29.249765 23487 tablet_copy_service.cc:342] P 5ee9b8f29d3349f4ba05425d5287ff58: Request end of tablet copy session 1a4466679fbc43618c8fb2e3d8a26dfd-3aba18dc398d4c8eb1dfce74850a1478 received from {username='slave'} at 127.20.240.67:51827
I20240503 14:12:29.249876 23487 tablet_copy_service.cc:433] P 5ee9b8f29d3349f4ba05425d5287ff58: ending tablet copy session 1a4466679fbc43618c8fb2e3d8a26dfd-3aba18dc398d4c8eb1dfce74850a1478 on tablet 3aba18dc398d4c8eb1dfce74850a1478 with peer 1a4466679fbc43618c8fb2e3d8a26dfd
I20240503 14:12:29.741645 24697 raft_consensus.cc:1203] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:29.742924 25593 raft_consensus.cc:2932] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:29.744580 24697 raft_consensus.cc:2932] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:29.748781 23326 catalog_manager.cc:5484] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: config changed from index -1 to 4, VOTER 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:29.791561 23467 consensus_queue.cc:237] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
W20240503 14:12:29.800714 23401 consensus_peers.cc:485] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 -> Peer 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69:38795): Couldn't send request to peer 4fbbe1e5ec864182b3116ef1278f7a3a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed49476f299e4e8793bdfee31165ec8b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:30.410420 25621 ts_tablet_manager.cc:902] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: Initiating tablet copy from peer 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:30.411566 25621 tablet_copy_client.cc:308] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:36725
I20240503 14:12:30.417424 23487 tablet_copy_service.cc:140] P 5ee9b8f29d3349f4ba05425d5287ff58: Received BeginTabletCopySession request for tablet ed49476f299e4e8793bdfee31165ec8b from peer 4fbbe1e5ec864182b3116ef1278f7a3a ({username='slave'} at 127.20.240.69:38035)
I20240503 14:12:30.417593 23487 tablet_copy_service.cc:161] P 5ee9b8f29d3349f4ba05425d5287ff58: Beginning new tablet copy session on tablet ed49476f299e4e8793bdfee31165ec8b from peer 4fbbe1e5ec864182b3116ef1278f7a3a at {username='slave'} at 127.20.240.69:38035: session id = 4fbbe1e5ec864182b3116ef1278f7a3a-ed49476f299e4e8793bdfee31165ec8b
I20240503 14:12:30.419592 23487 tablet_copy_source_session.cc:205] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:30.420698 25621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed49476f299e4e8793bdfee31165ec8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:30.424808 25621 tablet_copy_client.cc:783] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:30.425069 25621 tablet_copy_client.cc:647] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:30.426560 25621 tablet_copy_client.cc:523] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:30.428750 25621 tablet_bootstrap.cc:492] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: Bootstrap starting.
I20240503 14:12:30.457626 25621 tablet_bootstrap.cc:492] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:30.457959 25621 tablet_bootstrap.cc:492] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: Bootstrap complete.
I20240503 14:12:30.458170 25621 ts_tablet_manager.cc:1367] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20240503 14:12:30.458978 25621 raft_consensus.cc:348] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:30.459326 25621 raft_consensus.cc:729] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fbbe1e5ec864182b3116ef1278f7a3a, State: Initialized, Role: FOLLOWER
I20240503 14:12:30.459522 25621 consensus_queue.cc:260] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:30.460286 25621 ts_tablet_manager.cc:1398] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:30.461019 23487 tablet_copy_service.cc:342] P 5ee9b8f29d3349f4ba05425d5287ff58: Request end of tablet copy session 4fbbe1e5ec864182b3116ef1278f7a3a-ed49476f299e4e8793bdfee31165ec8b received from {username='slave'} at 127.20.240.69:38035
I20240503 14:12:30.461158 23487 tablet_copy_service.cc:433] P 5ee9b8f29d3349f4ba05425d5287ff58: ending tablet copy session 4fbbe1e5ec864182b3116ef1278f7a3a-ed49476f299e4e8793bdfee31165ec8b on tablet ed49476f299e4e8793bdfee31165ec8b with peer 4fbbe1e5ec864182b3116ef1278f7a3a
I20240503 14:12:30.975064 24968 raft_consensus.cc:1203] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.976429 25631 raft_consensus.cc:2932] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:30.977365 24968 raft_consensus.cc:2932] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:30.981411 25631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49888 (request call id 0) took 1190 ms. Trace:
I20240503 14:12:30.981619 25631 rpcz_store.cc:276] 0503 14:12:29.790847 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:29.790925 (+ 78us) service_pool.cc:225] Handling call
0503 14:12:29.792369 (+ 1444us) log.cc:842] Serialized 260 byte log entry
0503 14:12:30.981354 (+1188985us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":79,"raft.queue_time_us":48,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":223,"thread_start_us":209,"threads_started":1,"wal-append.queue_time_us":294,"wal-append.run_cpu_time_us":494,"wal-append.run_wall_time_us":1000556}
I20240503 14:12:30.982352 23326 catalog_manager.cc:5484] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a reported cstate change: config changed from index -1 to 4, VOTER 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:31.019788 23598 consensus_queue.cc:237] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
W20240503 14:12:31.026737 23533 consensus_peers.cc:485] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 -> Peer 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71:43179): Couldn't send request to peer 0e71aaeb321d4364bc1d81d64815b201. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4580e6da50584b93a7eab07537fa8a73. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:31.508390 25659 ts_tablet_manager.cc:902] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: Initiating tablet copy from peer 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:31.509608 25659 tablet_copy_client.cc:308] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:36169
I20240503 14:12:31.515733 23618 tablet_copy_service.cc:140] P 7f3556100c8d46aaafb47e87defe8064: Received BeginTabletCopySession request for tablet 4580e6da50584b93a7eab07537fa8a73 from peer 0e71aaeb321d4364bc1d81d64815b201 ({username='slave'} at 127.20.240.71:60825)
I20240503 14:12:31.515930 23618 tablet_copy_service.cc:161] P 7f3556100c8d46aaafb47e87defe8064: Beginning new tablet copy session on tablet 4580e6da50584b93a7eab07537fa8a73 from peer 0e71aaeb321d4364bc1d81d64815b201 at {username='slave'} at 127.20.240.71:60825: session id = 0e71aaeb321d4364bc1d81d64815b201-4580e6da50584b93a7eab07537fa8a73
I20240503 14:12:31.517820 23618 tablet_copy_source_session.cc:205] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:31.518990 25659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4580e6da50584b93a7eab07537fa8a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:31.523125 25659 tablet_copy_client.cc:783] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:31.523348 25659 tablet_copy_client.cc:647] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:31.524819 25659 tablet_copy_client.cc:523] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:31.527266 25659 tablet_bootstrap.cc:492] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: Bootstrap starting.
I20240503 14:12:31.555783 25659 tablet_bootstrap.cc:492] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:31.556140 25659 tablet_bootstrap.cc:492] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: Bootstrap complete.
I20240503 14:12:31.556349 25659 ts_tablet_manager.cc:1367] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240503 14:12:31.557051 25659 raft_consensus.cc:348] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:31.557359 25659 raft_consensus.cc:729] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e71aaeb321d4364bc1d81d64815b201, State: Initialized, Role: FOLLOWER
I20240503 14:12:31.557497 25659 consensus_queue.cc:260] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:31.558115 25659 ts_tablet_manager.cc:1398] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:12:31.558893 23618 tablet_copy_service.cc:342] P 7f3556100c8d46aaafb47e87defe8064: Request end of tablet copy session 0e71aaeb321d4364bc1d81d64815b201-4580e6da50584b93a7eab07537fa8a73 received from {username='slave'} at 127.20.240.71:60825
I20240503 14:12:31.559026 23618 tablet_copy_service.cc:433] P 7f3556100c8d46aaafb47e87defe8064: ending tablet copy session 0e71aaeb321d4364bc1d81d64815b201-4580e6da50584b93a7eab07537fa8a73 on tablet 4580e6da50584b93a7eab07537fa8a73 with peer 0e71aaeb321d4364bc1d81d64815b201
I20240503 14:12:31.895489 25229 raft_consensus.cc:1203] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:31.896862 25667 raft_consensus.cc:2932] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:31.897655 25229 raft_consensus.cc:2932] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:31.902810 23326 catalog_manager.cc:5484] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 reported cstate change: config changed from index -1 to 4, VOTER 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:32.023980 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.039525 25209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.043392 24677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.047622 23447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:32.053471 23578 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:32.092602 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.103462 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.147413 23598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4580e6da50584b93a7eab07537fa8a73"
dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32880
I20240503 14:12:32.147629 23598 raft_consensus.cc:595] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:32.233162 25209 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.236009 24677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.251968 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.270032 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.273164 23578 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:32.284444 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.297667 23447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:32.312954 25667 raft_consensus.cc:979] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: : Instructing follower 0e71aaeb321d4364bc1d81d64815b201 to start an election
I20240503 14:12:32.313108 25667 raft_consensus.cc:1067] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Signalling peer 0e71aaeb321d4364bc1d81d64815b201 to start an election
I20240503 14:12:32.314142 25229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4580e6da50584b93a7eab07537fa8a73"
dest_uuid: "0e71aaeb321d4364bc1d81d64815b201"
from {username='slave'} at 127.20.240.66:47069
I20240503 14:12:32.314424 25229 raft_consensus.cc:482] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:32.314589 25229 raft_consensus.cc:3037] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:32.317533 25229 raft_consensus.cc:504] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:32.319875 25229 leader_election.cc:288] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [CANDIDATE]: Term 2 election: Requested vote from peers 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:32.327073 23598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4580e6da50584b93a7eab07537fa8a73" candidate_uuid: "0e71aaeb321d4364bc1d81d64815b201" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
I20240503 14:12:32.327399 23598 raft_consensus.cc:3032] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:32.327497 23598 raft_consensus.cc:729] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:32.327770 23598 consensus_queue.cc:260] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:32.328035 23598 raft_consensus.cc:3037] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:32.331384 23598 raft_consensus.cc:2454] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e71aaeb321d4364bc1d81d64815b201 in term 2.
I20240503 14:12:32.332006 25165 leader_election.cc:302] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e71aaeb321d4364bc1d81d64815b201, 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:32.332592 25666 raft_consensus.cc:2781] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:32.337858 25666 raft_consensus.cc:686] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 LEADER]: Becoming Leader. State: Replica: 0e71aaeb321d4364bc1d81d64815b201, State: Running, Role: LEADER
I20240503 14:12:32.338505 25666 consensus_queue.cc:237] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:32.342141 23323 catalog_manager.cc:5484] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 reported cstate change: term changed from 1 to 2, leader changed from 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) to 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71). New cstate: current_term: 2 leader_uuid: "0e71aaeb321d4364bc1d81d64815b201" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:32.464154 24677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.464462 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.472638 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.473394 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.489480 23447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:32.495218 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:32.522495 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.580905 23467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3aba18dc398d4c8eb1dfce74850a1478"
dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49922
I20240503 14:12:32.581185 23467 raft_consensus.cc:595] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:32.684324 25632 raft_consensus.cc:979] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: : Instructing follower 1a4466679fbc43618c8fb2e3d8a26dfd to start an election
I20240503 14:12:32.684545 25632 raft_consensus.cc:1067] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Signalling peer 1a4466679fbc43618c8fb2e3d8a26dfd to start an election
I20240503 14:12:32.692257 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.696638 24677 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.707695 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:32.710603 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.711217 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.719713 23447 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:32.734478 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:12:32.744760 23402 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:43179, user_credentials={real_user=slave}} blocked reactor thread for 64436.4us
I20240503 14:12:32.745817 24697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3aba18dc398d4c8eb1dfce74850a1478"
dest_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd"
from {username='slave'} at 127.20.240.65:33665
I20240503 14:12:32.746475 24697 raft_consensus.cc:482] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:32.746870 24697 raft_consensus.cc:3037] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:32.749790 24697 raft_consensus.cc:504] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:32.751883 23467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aba18dc398d4c8eb1dfce74850a1478" candidate_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
I20240503 14:12:32.752174 23467 raft_consensus.cc:3032] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:32.752254 23467 raft_consensus.cc:729] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:32.752543 23467 consensus_queue.cc:260] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:32.752789 23467 raft_consensus.cc:3037] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:32.756026 23467 raft_consensus.cc:2454] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a4466679fbc43618c8fb2e3d8a26dfd in term 2.
I20240503 14:12:32.756676 24630 leader_election.cc:302] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1a4466679fbc43618c8fb2e3d8a26dfd, 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:32.758744 24697 leader_election.cc:288] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [CANDIDATE]: Term 2 election: Requested vote from peers 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:32.766976 25838 raft_consensus.cc:2781] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:32.774147 25838 raft_consensus.cc:686] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 LEADER]: Becoming Leader. State: Replica: 1a4466679fbc43618c8fb2e3d8a26dfd, State: Running, Role: LEADER
I20240503 14:12:32.774863 25838 consensus_queue.cc:237] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:32.777112 23598 raft_consensus.cc:1261] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Refusing update from remote peer 0e71aaeb321d4364bc1d81d64815b201: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:32.778445 25666 consensus_queue.cc:1035] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:32.786635 23326 catalog_manager.cc:5484] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd reported cstate change: term changed from 1 to 2, leader changed from 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) to 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67). New cstate: current_term: 2 leader_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:32.933315 24677 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.933547 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.943181 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:32.960704 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:32.967067 23447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:32.986232 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:32.989158 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.051394 23467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d433f9489b204c8f9e1ef5c2164c5a6e"
dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49952
I20240503 14:12:33.051590 23467 raft_consensus.cc:595] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:33.155977 24677 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.156666 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.158933 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.159005 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.181849 23447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:33.194922 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:33.219027 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.239343 23467 raft_consensus.cc:1261] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Refusing update from remote peer 1a4466679fbc43618c8fb2e3d8a26dfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:33.240386 25838 consensus_queue.cc:1035] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:33.355326 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.357898 24677 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.361979 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.385327 23447 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:33.396230 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.419505 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:33.441512 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.492337 23467 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed49476f299e4e8793bdfee31165ec8b"
dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49986
I20240503 14:12:33.492535 23467 raft_consensus.cc:595] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:33.518185 25835 raft_consensus.cc:979] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: : Instructing follower 1a4466679fbc43618c8fb2e3d8a26dfd to start an election
I20240503 14:12:33.518393 25835 raft_consensus.cc:1067] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Signalling peer 1a4466679fbc43618c8fb2e3d8a26dfd to start an election
I20240503 14:12:33.519546 24697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d433f9489b204c8f9e1ef5c2164c5a6e"
dest_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd"
from {username='slave'} at 127.20.240.65:33665
I20240503 14:12:33.519804 24697 raft_consensus.cc:482] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:33.519945 24697 raft_consensus.cc:3037] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:33.522667 24697 raft_consensus.cc:504] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:33.523169 24697 leader_election.cc:288] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [CANDIDATE]: Term 2 election: Requested vote from peers 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:33.523658 23467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d433f9489b204c8f9e1ef5c2164c5a6e" candidate_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
I20240503 14:12:33.523840 23467 raft_consensus.cc:3032] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:33.523931 23467 raft_consensus.cc:729] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:33.524196 23467 consensus_queue.cc:260] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:33.524464 23467 raft_consensus.cc:3037] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:33.526448 23467 raft_consensus.cc:2454] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a4466679fbc43618c8fb2e3d8a26dfd in term 2.
I20240503 14:12:33.526795 24630 leader_election.cc:302] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 1a4466679fbc43618c8fb2e3d8a26dfd, 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:33.527073 25838 raft_consensus.cc:2781] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:33.527221 25838 raft_consensus.cc:686] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 LEADER]: Becoming Leader. State: Replica: 1a4466679fbc43618c8fb2e3d8a26dfd, State: Running, Role: LEADER
I20240503 14:12:33.527488 25838 consensus_queue.cc:237] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:33.529760 23325 catalog_manager.cc:5484] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd reported cstate change: term changed from 1 to 2, leader changed from 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) to 1a4466679fbc43618c8fb2e3d8a26dfd (127.20.240.67). New cstate: current_term: 2 leader_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:33.593351 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.601378 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.607606 23447 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:33.614158 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.624879 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.647219 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:33.648068 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.813638 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.827704 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:33.836470 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.846558 25975 raft_consensus.cc:979] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: : Instructing follower 4fbbe1e5ec864182b3116ef1278f7a3a to start an election
I20240503 14:12:33.846740 25975 raft_consensus.cc:1067] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Signalling peer 4fbbe1e5ec864182b3116ef1278f7a3a to start an election
I20240503 14:12:33.847433 23447 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:33.847954 24968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed49476f299e4e8793bdfee31165ec8b"
dest_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a"
from {username='slave'} at 127.20.240.65:52159
I20240503 14:12:33.848254 24968 raft_consensus.cc:482] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:33.848453 24968 raft_consensus.cc:3037] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:33.851653 24968 raft_consensus.cc:504] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:33.853534 24968 leader_election.cc:288] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [CANDIDATE]: Term 2 election: Requested vote from peers 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:33.854156 23467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed49476f299e4e8793bdfee31165ec8b" candidate_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
I20240503 14:12:33.854378 23467 raft_consensus.cc:3032] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:33.854470 23467 raft_consensus.cc:729] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:33.854713 23467 consensus_queue.cc:260] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:33.854952 23467 raft_consensus.cc:3037] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:33.856761 23467 raft_consensus.cc:2454] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fbbe1e5ec864182b3116ef1278f7a3a in term 2.
I20240503 14:12:33.857437 24902 leader_election.cc:302] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4fbbe1e5ec864182b3116ef1278f7a3a, 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:33.858054 26043 raft_consensus.cc:2781] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:33.862628 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:33.876268 24949 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.878372 26043 raft_consensus.cc:686] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 LEADER]: Becoming Leader. State: Replica: 4fbbe1e5ec864182b3116ef1278f7a3a, State: Running, Role: LEADER
I20240503 14:12:33.878962 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:33.879082 26043 consensus_queue.cc:237] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:33.890451 23323 catalog_manager.cc:5484] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a reported cstate change: term changed from 1 to 2, leader changed from 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) to 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69). New cstate: current_term: 2 leader_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:33.951612 23598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "feacaf3f7e794a19b20a4dea58730842"
dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32972
I20240503 14:12:33.951807 23598 raft_consensus.cc:595] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:34.043397 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:34.045640 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.048328 24949 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.059362 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.070999 23447 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:34.077664 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:34.083053 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:34.094296 23467 raft_consensus.cc:1261] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Refusing update from remote peer 1a4466679fbc43618c8fb2e3d8a26dfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:34.095010 25838 consensus_queue.cc:1035] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:34.259215 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.272812 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:34.276028 24949 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.277140 23447 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:34.286952 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.320147 23578 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:12:34.332443 23467 raft_consensus.cc:1261] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Refusing update from remote peer 4fbbe1e5ec864182b3116ef1278f7a3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:34.334038 26043 consensus_queue.cc:1035] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:12:34.338037 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:34.408812 23598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54ff6aaa60734f548ff2b8da41ea5b3a"
dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32998
I20240503 14:12:34.409044 23598 raft_consensus.cc:595] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:34.486634 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.490697 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:34.505313 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.511893 24949 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.521780 23447 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:34.536129 26161 raft_consensus.cc:979] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: : Instructing follower 95dfe9d1f79f4a8696fc5586e3ea3731 to start an election
I20240503 14:12:34.536379 25620 raft_consensus.cc:1067] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Signalling peer 95dfe9d1f79f4a8696fc5586e3ea3731 to start an election
I20240503 14:12:34.538766 25099 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "feacaf3f7e794a19b20a4dea58730842"
dest_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731"
from {username='slave'} at 127.20.240.66:55443
I20240503 14:12:34.539124 25099 raft_consensus.cc:482] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:34.547806 25099 raft_consensus.cc:3037] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:34.559883 25099 raft_consensus.cc:504] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:34.561332 25099 leader_election.cc:288] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [CANDIDATE]: Term 2 election: Requested vote from peers 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:34.570077 23598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feacaf3f7e794a19b20a4dea58730842" candidate_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
I20240503 14:12:34.570336 23598 raft_consensus.cc:3032] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:34.570451 23598 raft_consensus.cc:729] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:34.570699 23598 consensus_queue.cc:260] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:34.571007 23598 raft_consensus.cc:3037] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:34.573730 23598 raft_consensus.cc:2454] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95dfe9d1f79f4a8696fc5586e3ea3731 in term 2.
I20240503 14:12:34.578433 25035 leader_election.cc:302] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064, 95dfe9d1f79f4a8696fc5586e3ea3731; no voters:
I20240503 14:12:34.580291 23578 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:34.582453 25079 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:34.586656 26162 raft_consensus.cc:2781] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:34.594416 25620 raft_consensus.cc:979] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: : Instructing follower 4fbbe1e5ec864182b3116ef1278f7a3a to start an election
I20240503 14:12:34.594550 25620 raft_consensus.cc:1067] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Signalling peer 4fbbe1e5ec864182b3116ef1278f7a3a to start an election
I20240503 14:12:34.595235 24968 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54ff6aaa60734f548ff2b8da41ea5b3a"
dest_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a"
from {username='slave'} at 127.20.240.66:54307
I20240503 14:12:34.595453 24968 raft_consensus.cc:482] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:34.595551 24968 raft_consensus.cc:3037] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:34.598662 24968 raft_consensus.cc:504] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:34.605509 26162 raft_consensus.cc:686] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 LEADER]: Becoming Leader. State: Replica: 95dfe9d1f79f4a8696fc5586e3ea3731, State: Running, Role: LEADER
I20240503 14:12:34.606174 26162 consensus_queue.cc:237] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:34.609885 24968 leader_election.cc:288] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [CANDIDATE]: Term 2 election: Requested vote from peers 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:34.610867 23598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54ff6aaa60734f548ff2b8da41ea5b3a" candidate_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
I20240503 14:12:34.611073 23598 raft_consensus.cc:3032] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:34.611140 23598 raft_consensus.cc:729] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:34.611371 23598 consensus_queue.cc:260] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:34.611667 23598 raft_consensus.cc:3037] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:34.615129 23598 raft_consensus.cc:2454] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fbbe1e5ec864182b3116ef1278f7a3a in term 2.
I20240503 14:12:34.615564 24905 leader_election.cc:302] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4fbbe1e5ec864182b3116ef1278f7a3a, 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:34.615991 26043 raft_consensus.cc:2781] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:34.616132 26043 raft_consensus.cc:686] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 LEADER]: Becoming Leader. State: Replica: 4fbbe1e5ec864182b3116ef1278f7a3a, State: Running, Role: LEADER
I20240503 14:12:34.615967 23325 catalog_manager.cc:5484] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 reported cstate change: term changed from 1 to 2, leader changed from 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) to 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70). New cstate: current_term: 2 leader_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:34.616384 26043 consensus_queue.cc:237] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:34.628854 23326 catalog_manager.cc:5484] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a reported cstate change: term changed from 1 to 2, leader changed from 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) to 4fbbe1e5ec864182b3116ef1278f7a3a (127.20.240.69). New cstate: current_term: 2 leader_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:34.770175 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.773417 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:34.780668 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.789654 23447 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:34.790390 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:34.810572 23578 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:34.822234 25079 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.881172 23466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "76c35354470b4d2187c8dd6c0d1594f4"
dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50072
I20240503 14:12:34.881387 23466 raft_consensus.cc:595] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:34.961443 25209 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:34.962802 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:34.979286 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:35.008620 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.022020 23447 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:12:35.025468 23578 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:35.026953 23598 raft_consensus.cc:1261] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Refusing update from remote peer 95dfe9d1f79f4a8696fc5586e3ea3731: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:35.027760 26162 consensus_queue.cc:1035] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:35.034160 25079 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:35.053257 26252 raft_consensus.cc:979] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: : Instructing follower 0e71aaeb321d4364bc1d81d64815b201 to start an election
I20240503 14:12:35.060065 25975 raft_consensus.cc:1067] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Signalling peer 0e71aaeb321d4364bc1d81d64815b201 to start an election
I20240503 14:12:35.060842 25229 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "76c35354470b4d2187c8dd6c0d1594f4"
dest_uuid: "0e71aaeb321d4364bc1d81d64815b201"
from {username='slave'} at 127.20.240.65:57379
I20240503 14:12:35.061089 25229 raft_consensus.cc:482] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:35.061187 25229 raft_consensus.cc:3037] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:35.064023 25229 raft_consensus.cc:504] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:35.064657 25229 leader_election.cc:288] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [CANDIDATE]: Term 2 election: Requested vote from peers 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:35.065404 23466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c35354470b4d2187c8dd6c0d1594f4" candidate_uuid: "0e71aaeb321d4364bc1d81d64815b201" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
I20240503 14:12:35.066375 23466 raft_consensus.cc:3032] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:35.066483 23466 raft_consensus.cc:729] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:35.066762 23466 consensus_queue.cc:260] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:35.067055 23466 raft_consensus.cc:3037] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:35.069870 23466 raft_consensus.cc:2454] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e71aaeb321d4364bc1d81d64815b201 in term 2.
I20240503 14:12:35.070399 25162 leader_election.cc:302] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0e71aaeb321d4364bc1d81d64815b201, 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:35.070798 26046 raft_consensus.cc:2781] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:35.071223 26046 raft_consensus.cc:686] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 LEADER]: Becoming Leader. State: Replica: 0e71aaeb321d4364bc1d81d64815b201, State: Running, Role: LEADER
I20240503 14:12:35.071530 26046 consensus_queue.cc:237] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:35.080090 23326 catalog_manager.cc:5484] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 reported cstate change: term changed from 1 to 2, leader changed from 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) to 0e71aaeb321d4364bc1d81d64815b201 (127.20.240.71). New cstate: current_term: 2 leader_uuid: "0e71aaeb321d4364bc1d81d64815b201" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:35.176506 23597 raft_consensus.cc:1261] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Refusing update from remote peer 4fbbe1e5ec864182b3116ef1278f7a3a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:35.177059 26043 consensus_queue.cc:1035] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:35.229126 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.231634 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.238327 24809 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:35.253857 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.266166 23447 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:35.266849 23578 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:35.292562 25079 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:35.351306 23466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60a2f6744aa741d9b2af7eeefe868184"
dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50100
I20240503 14:12:35.351497 23466 raft_consensus.cc:595] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:35.353190 25975 raft_consensus.cc:979] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: : Instructing follower 2062470ed9164d389aa2e60973ecdcdb to start an election
I20240503 14:12:35.353336 25975 raft_consensus.cc:1067] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Signalling peer 2062470ed9164d389aa2e60973ecdcdb to start an election
I20240503 14:12:35.354373 24828 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60a2f6744aa741d9b2af7eeefe868184"
dest_uuid: "2062470ed9164d389aa2e60973ecdcdb"
from {username='slave'} at 127.20.240.65:50275
I20240503 14:12:35.354641 24828 raft_consensus.cc:482] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:35.354792 24828 raft_consensus.cc:3037] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:35.357079 24828 raft_consensus.cc:504] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:35.358270 24828 leader_election.cc:288] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [CANDIDATE]: Term 2 election: Requested vote from peers 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:35.358695 23466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a2f6744aa741d9b2af7eeefe868184" candidate_uuid: "2062470ed9164d389aa2e60973ecdcdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
I20240503 14:12:35.358901 23466 raft_consensus.cc:3032] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:35.358983 23466 raft_consensus.cc:729] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:35.359234 23466 consensus_queue.cc:260] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:35.359537 23466 raft_consensus.cc:3037] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:35.361444 23466 raft_consensus.cc:2454] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2062470ed9164d389aa2e60973ecdcdb in term 2.
I20240503 14:12:35.362021 24761 leader_election.cc:302] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2062470ed9164d389aa2e60973ecdcdb, 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:35.367966 26317 raft_consensus.cc:2781] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:35.368585 26317 raft_consensus.cc:686] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 2 LEADER]: Becoming Leader. State: Replica: 2062470ed9164d389aa2e60973ecdcdb, State: Running, Role: LEADER
I20240503 14:12:35.369004 26317 consensus_queue.cc:237] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:35.373062 23323 catalog_manager.cc:5484] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb reported cstate change: term changed from 1 to 2, leader changed from 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) to 2062470ed9164d389aa2e60973ecdcdb (127.20.240.68). New cstate: current_term: 2 leader_uuid: "2062470ed9164d389aa2e60973ecdcdb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:35.449676 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.449899 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.463469 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.472260 25079 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:35.475188 23578 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:35.476230 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.487334 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:35.639308 23466 raft_consensus.cc:1261] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Refusing update from remote peer 0e71aaeb321d4364bc1d81d64815b201: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:35.640193 26046 consensus_queue.cc:1035] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:35.684293 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.686731 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.697896 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.704267 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.709581 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:35.722602 23578 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:12:35.731201 25079 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:35.793195 23597 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a71476f51de74a25ba84fc92ef0f9414"
dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33070
I20240503 14:12:35.793408 23597 raft_consensus.cc:595] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:35.842012 23466 raft_consensus.cc:1261] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Refusing update from remote peer 2062470ed9164d389aa2e60973ecdcdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:35.842711 26317 consensus_queue.cc:1035] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:35.872097 26418 raft_consensus.cc:979] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: : Instructing follower 95dfe9d1f79f4a8696fc5586e3ea3731 to start an election
I20240503 14:12:35.872313 26414 raft_consensus.cc:1067] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Signalling peer 95dfe9d1f79f4a8696fc5586e3ea3731 to start an election
I20240503 14:12:35.873807 25099 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a71476f51de74a25ba84fc92ef0f9414"
dest_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731"
from {username='slave'} at 127.20.240.66:55443
I20240503 14:12:35.874008 25099 raft_consensus.cc:482] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:35.874119 25099 raft_consensus.cc:3037] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:35.877756 25099 raft_consensus.cc:504] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:35.878867 25099 leader_election.cc:288] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [CANDIDATE]: Term 2 election: Requested vote from peers 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:35.880270 23597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a71476f51de74a25ba84fc92ef0f9414" candidate_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
I20240503 14:12:35.880591 23597 raft_consensus.cc:3032] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:35.880681 23597 raft_consensus.cc:729] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:35.881011 23597 consensus_queue.cc:260] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:35.881361 23597 raft_consensus.cc:3037] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:35.884943 23597 raft_consensus.cc:2454] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95dfe9d1f79f4a8696fc5586e3ea3731 in term 2.
I20240503 14:12:35.886422 25035 leader_election.cc:302] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 7f3556100c8d46aaafb47e87defe8064, 95dfe9d1f79f4a8696fc5586e3ea3731; no voters:
I20240503 14:12:35.886798 26162 raft_consensus.cc:2781] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:35.887205 26162 raft_consensus.cc:686] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 LEADER]: Becoming Leader. State: Replica: 95dfe9d1f79f4a8696fc5586e3ea3731, State: Running, Role: LEADER
I20240503 14:12:35.887522 26162 consensus_queue.cc:237] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:35.891829 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.893893 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.902966 23325 catalog_manager.cc:5484] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 reported cstate change: term changed from 1 to 2, leader changed from 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) to 95dfe9d1f79f4a8696fc5586e3ea3731 (127.20.240.70). New cstate: current_term: 2 leader_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:35.905032 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.918138 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.918943 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:35.934273 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.978482 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:36.339010 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:36.348845 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.348870 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:36.360608 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.362970 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.374442 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.385406 23598 raft_consensus.cc:1261] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Refusing update from remote peer 95dfe9d1f79f4a8696fc5586e3ea3731: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:36.386101 26162 consensus_queue.cc:1035] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:36.391047 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.475926 25229 consensus_queue.cc:237] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:36.477116 26374 raft_consensus.cc:2932] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:36.481953 23325 catalog_manager.cc:5484] T 4580e6da50584b93a7eab07537fa8a73 P 0e71aaeb321d4364bc1d81d64815b201 reported cstate change: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New cstate: current_term: 2 leader_uuid: "0e71aaeb321d4364bc1d81d64815b201" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:36.495508 23578 tablet_service.cc:1514] Processing DeleteTablet for tablet 4580e6da50584b93a7eab07537fa8a73 with delete_type TABLET_DATA_TOMBSTONED (TS 7f3556100c8d46aaafb47e87defe8064 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37200
I20240503 14:12:36.496629 26498 tablet_replica.cc:330] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: stopping tablet replica
I20240503 14:12:36.497256 26498 raft_consensus.cc:2229] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:36.497565 26498 raft_consensus.cc:2258] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:36.499034 26498 ts_tablet_manager.cc:1875] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:36.505193 26498 ts_tablet_manager.cc:1888] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:36.505344 26498 log.cc:1196] T 4580e6da50584b93a7eab07537fa8a73 P 7f3556100c8d46aaafb47e87defe8064: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/4580e6da50584b93a7eab07537fa8a73
I20240503 14:12:36.506287 23313 catalog_manager.cc:4830] TS 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169): tablet 4580e6da50584b93a7eab07537fa8a73 (table rebalance_test_table_1 [id=78b403f6ecf94c72ada7ac48f4cc6957]) successfully deleted
I20240503 14:12:36.588769 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.589305 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.607864 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.617587 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.639273 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:36.658221 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:36.689664 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.765481 24697 consensus_queue.cc:237] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:36.766800 26300 raft_consensus.cc:2932] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:36.771842 23325 catalog_manager.cc:5484] T 3aba18dc398d4c8eb1dfce74850a1478 P 1a4466679fbc43618c8fb2e3d8a26dfd reported cstate change: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:36.775506 23447 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aba18dc398d4c8eb1dfce74850a1478 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee9b8f29d3349f4ba05425d5287ff58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53126
I20240503 14:12:36.776329 26550 tablet_replica.cc:330] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: stopping tablet replica
I20240503 14:12:36.776782 26550 raft_consensus.cc:2229] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:36.777101 26550 raft_consensus.cc:2258] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:36.778180 26550 ts_tablet_manager.cc:1875] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:36.782624 26550 ts_tablet_manager.cc:1888] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:36.782730 26550 log.cc:1196] T 3aba18dc398d4c8eb1dfce74850a1478 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/3aba18dc398d4c8eb1dfce74850a1478
I20240503 14:12:36.783525 23310 catalog_manager.cc:4830] TS 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725): tablet 3aba18dc398d4c8eb1dfce74850a1478 (table rebalance_test_table_0 [id=d9694e7b92ac404db0c54d5ee5dfc86d]) successfully deleted
I20240503 14:12:36.883611 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.892177 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.895283 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.897101 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:36.899478 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:36.948431 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:36.961982 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.024024 24697 consensus_queue.cc:237] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } }
I20240503 14:12:37.025206 26300 raft_consensus.cc:2932] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:37.029328 23325 catalog_manager.cc:5484] T d433f9489b204c8f9e1ef5c2164c5a6e P 1a4466679fbc43618c8fb2e3d8a26dfd reported cstate change: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1a4466679fbc43618c8fb2e3d8a26dfd" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 41087 } } }
I20240503 14:12:37.033511 23447 tablet_service.cc:1514] Processing DeleteTablet for tablet d433f9489b204c8f9e1ef5c2164c5a6e with delete_type TABLET_DATA_TOMBSTONED (TS 5ee9b8f29d3349f4ba05425d5287ff58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53126
I20240503 14:12:37.033872 26550 tablet_replica.cc:330] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: stopping tablet replica
I20240503 14:12:37.034092 26550 raft_consensus.cc:2229] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:37.034324 26550 raft_consensus.cc:2258] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:37.034929 26550 ts_tablet_manager.cc:1875] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:37.039542 26550 ts_tablet_manager.cc:1888] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:37.039681 26550 log.cc:1196] T d433f9489b204c8f9e1ef5c2164c5a6e P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/d433f9489b204c8f9e1ef5c2164c5a6e
I20240503 14:12:37.040475 23310 catalog_manager.cc:4830] TS 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725): tablet d433f9489b204c8f9e1ef5c2164c5a6e (table rebalance_test_table_2 [id=13c93033066d40228c6279535bf62553]) successfully deleted
I20240503 14:12:37.127328 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.131356 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.140240 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.140812 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.141680 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.141886 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.174302 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.258292 24968 consensus_queue.cc:237] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:37.259634 26289 raft_consensus.cc:2932] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:37.263753 23325 catalog_manager.cc:5484] T ed49476f299e4e8793bdfee31165ec8b P 4fbbe1e5ec864182b3116ef1278f7a3a reported cstate change: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:37.267105 23447 tablet_service.cc:1514] Processing DeleteTablet for tablet ed49476f299e4e8793bdfee31165ec8b with delete_type TABLET_DATA_TOMBSTONED (TS 5ee9b8f29d3349f4ba05425d5287ff58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53126
I20240503 14:12:37.267431 26550 tablet_replica.cc:330] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: stopping tablet replica
I20240503 14:12:37.267647 26550 raft_consensus.cc:2229] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:37.267858 26550 raft_consensus.cc:2258] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:37.268488 26550 ts_tablet_manager.cc:1875] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:37.272817 26550 ts_tablet_manager.cc:1888] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:37.272945 26550 log.cc:1196] T ed49476f299e4e8793bdfee31165ec8b P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed49476f299e4e8793bdfee31165ec8b
I20240503 14:12:37.273646 23310 catalog_manager.cc:4830] TS 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725): tablet ed49476f299e4e8793bdfee31165ec8b (table rebalance_test_table_3 [id=14847a692cab4bf487630962c7ba1934]) successfully deleted
I20240503 14:12:37.376009 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.391618 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.391765 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.395910 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.399163 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.415390 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.457131 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.538148 25099 consensus_queue.cc:237] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:37.539441 26470 raft_consensus.cc:2932] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:37.544036 23323 catalog_manager.cc:5484] T feacaf3f7e794a19b20a4dea58730842 P 95dfe9d1f79f4a8696fc5586e3ea3731 reported cstate change: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New cstate: current_term: 2 leader_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:37.547672 23578 tablet_service.cc:1514] Processing DeleteTablet for tablet feacaf3f7e794a19b20a4dea58730842 with delete_type TABLET_DATA_TOMBSTONED (TS 7f3556100c8d46aaafb47e87defe8064 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37200
I20240503 14:12:37.548381 26715 tablet_replica.cc:330] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: stopping tablet replica
I20240503 14:12:37.548590 26715 raft_consensus.cc:2229] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:37.548830 26715 raft_consensus.cc:2258] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:37.549476 26715 ts_tablet_manager.cc:1875] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:37.555835 26715 ts_tablet_manager.cc:1888] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:37.555958 26715 log.cc:1196] T feacaf3f7e794a19b20a4dea58730842 P 7f3556100c8d46aaafb47e87defe8064: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/feacaf3f7e794a19b20a4dea58730842
I20240503 14:12:37.556723 23313 catalog_manager.cc:4830] TS 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169): tablet feacaf3f7e794a19b20a4dea58730842 (table rebalance_test_table_3 [id=14847a692cab4bf487630962c7ba1934]) successfully deleted
I20240503 14:12:37.667367 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.669330 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.669351 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.707367 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.708068 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.708381 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.729287 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.798619 24968 consensus_queue.cc:237] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } }
I20240503 14:12:37.799695 26289 raft_consensus.cc:2932] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:37.803889 23325 catalog_manager.cc:5484] T 54ff6aaa60734f548ff2b8da41ea5b3a P 4fbbe1e5ec864182b3116ef1278f7a3a reported cstate change: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New cstate: current_term: 2 leader_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4fbbe1e5ec864182b3116ef1278f7a3a" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 38795 } } }
I20240503 14:12:37.807611 23578 tablet_service.cc:1514] Processing DeleteTablet for tablet 54ff6aaa60734f548ff2b8da41ea5b3a with delete_type TABLET_DATA_TOMBSTONED (TS 7f3556100c8d46aaafb47e87defe8064 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37200
I20240503 14:12:37.807953 26715 tablet_replica.cc:330] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: stopping tablet replica
I20240503 14:12:37.808153 26715 raft_consensus.cc:2229] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:37.808364 26715 raft_consensus.cc:2258] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:37.808951 26715 ts_tablet_manager.cc:1875] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:37.814095 26715 ts_tablet_manager.cc:1888] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:37.814322 26715 log.cc:1196] T 54ff6aaa60734f548ff2b8da41ea5b3a P 7f3556100c8d46aaafb47e87defe8064: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/54ff6aaa60734f548ff2b8da41ea5b3a
I20240503 14:12:37.815320 23313 catalog_manager.cc:4830] TS 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169): tablet 54ff6aaa60734f548ff2b8da41ea5b3a (table rebalance_test_table_1 [id=78b403f6ecf94c72ada7ac48f4cc6957]) successfully deleted
I20240503 14:12:37.905543 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.909760 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.918864 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.919588 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.920776 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:37.929555 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:37.951392 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.041206 25229 consensus_queue.cc:237] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } }
I20240503 14:12:38.042492 26801 raft_consensus.cc:2932] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:38.047694 23325 catalog_manager.cc:5484] T 76c35354470b4d2187c8dd6c0d1594f4 P 0e71aaeb321d4364bc1d81d64815b201 reported cstate change: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "0e71aaeb321d4364bc1d81d64815b201" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0e71aaeb321d4364bc1d81d64815b201" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 43179 } } }
I20240503 14:12:38.051816 23447 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c35354470b4d2187c8dd6c0d1594f4 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee9b8f29d3349f4ba05425d5287ff58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53126
I20240503 14:12:38.052502 26824 tablet_replica.cc:330] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: stopping tablet replica
I20240503 14:12:38.052695 26824 raft_consensus.cc:2229] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:38.052896 26824 raft_consensus.cc:2258] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:38.053630 26824 ts_tablet_manager.cc:1875] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:38.059115 26824 ts_tablet_manager.cc:1888] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:38.059326 26824 log.cc:1196] T 76c35354470b4d2187c8dd6c0d1594f4 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/76c35354470b4d2187c8dd6c0d1594f4
I20240503 14:12:38.060190 23310 catalog_manager.cc:4830] TS 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725): tablet 76c35354470b4d2187c8dd6c0d1594f4 (table rebalance_test_table_4 [id=efb3d2e243c246a393e534a59a399d94]) successfully deleted
I20240503 14:12:38.143270 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.145072 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.154181 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.171720 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:38.201027 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:38.202034 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.212474 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.289598 24828 consensus_queue.cc:237] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } }
I20240503 14:12:38.290858 26831 raft_consensus.cc:2932] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:38.295485 23325 catalog_manager.cc:5484] T 60a2f6744aa741d9b2af7eeefe868184 P 2062470ed9164d389aa2e60973ecdcdb reported cstate change: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "2062470ed9164d389aa2e60973ecdcdb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2062470ed9164d389aa2e60973ecdcdb" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 44039 } } }
I20240503 14:12:38.299420 23447 tablet_service.cc:1514] Processing DeleteTablet for tablet 60a2f6744aa741d9b2af7eeefe868184 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee9b8f29d3349f4ba05425d5287ff58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53126
I20240503 14:12:38.299794 26824 tablet_replica.cc:330] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: stopping tablet replica
I20240503 14:12:38.299988 26824 raft_consensus.cc:2229] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:38.300181 26824 raft_consensus.cc:2258] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:38.300762 26824 ts_tablet_manager.cc:1875] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:38.305784 26824 ts_tablet_manager.cc:1888] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:38.305912 26824 log.cc:1196] T 60a2f6744aa741d9b2af7eeefe868184 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/60a2f6744aa741d9b2af7eeefe868184
I20240503 14:12:38.306684 23310 catalog_manager.cc:4830] TS 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725): tablet 60a2f6744aa741d9b2af7eeefe868184 (table rebalance_test_table_4 [id=efb3d2e243c246a393e534a59a399d94]) successfully deleted
I20240503 14:12:38.414986 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.415627 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.424116 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.428483 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:38.431419 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:38.431468 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.461091 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.544077 25099 consensus_queue.cc:237] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } }
I20240503 14:12:38.545574 26832 raft_consensus.cc:2932] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:38.551700 23326 catalog_manager.cc:5484] T a71476f51de74a25ba84fc92ef0f9414 P 95dfe9d1f79f4a8696fc5586e3ea3731 reported cstate change: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New cstate: current_term: 2 leader_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "95dfe9d1f79f4a8696fc5586e3ea3731" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 34091 } } }
I20240503 14:12:38.554801 23578 tablet_service.cc:1514] Processing DeleteTablet for tablet a71476f51de74a25ba84fc92ef0f9414 with delete_type TABLET_DATA_TOMBSTONED (TS 7f3556100c8d46aaafb47e87defe8064 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37200
I20240503 14:12:38.555382 26928 tablet_replica.cc:330] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: stopping tablet replica
I20240503 14:12:38.555562 26928 raft_consensus.cc:2229] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:38.555761 26928 raft_consensus.cc:2258] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:38.556311 26928 ts_tablet_manager.cc:1875] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:38.560631 26928 ts_tablet_manager.cc:1888] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:38.560739 26928 log.cc:1196] T a71476f51de74a25ba84fc92ef0f9414 P 7f3556100c8d46aaafb47e87defe8064: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/a71476f51de74a25ba84fc92ef0f9414
I20240503 14:12:38.561457 23313 catalog_manager.cc:4830] TS 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169): tablet a71476f51de74a25ba84fc92ef0f9414 (table rebalance_test_table_2 [id=13c93033066d40228c6279535bf62553]) successfully deleted
I20240503 14:12:38.917814 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.920126 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.927178 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.946545 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.957414 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:38.988636 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:39.007295 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.147017 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.158303 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:39.159396 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.164350 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.173600 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.176790 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:39.199151 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.914746 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.919308 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.936188 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.955914 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.966034 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:39.990648 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:40.003520 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.183938 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.184340 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.193279 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:40.197016 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.207067 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:40.210796 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.214792 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.490451 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.491263 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.503026 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.519086 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:40.553130 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:40.568192 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:40.591279 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4c8d95e4965f4cb2bfdbc452fe5f7e0b | 127.20.240.126:43899 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.240.84:40571 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0e71aaeb321d4364bc1d81d64815b201 | 127.20.240.71:43179 | HEALTHY | <none> | 2 | 0
1a4466679fbc43618c8fb2e3d8a26dfd | 127.20.240.67:41087 | HEALTHY | <none> | 2 | 0
2062470ed9164d389aa2e60973ecdcdb | 127.20.240.68:44039 | HEALTHY | <none> | 2 | 0
4fbbe1e5ec864182b3116ef1278f7a3a | 127.20.240.69:38795 | HEALTHY | <none> | 2 | 0
5ee9b8f29d3349f4ba05425d5287ff58 | 127.20.240.65:36725 | HEALTHY | <none> | 3 | 0
7f3556100c8d46aaafb47e87defe8064 | 127.20.240.66:36169 | HEALTHY | <none> | 3 | 0
95dfe9d1f79f4a8696fc5586e3ea3731 | 127.20.240.70:34091 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.20.240.65 | experimental | 127.20.240.65:36725
local_ip_for_outbound_sockets | 127.20.240.66 | experimental | 127.20.240.66:36169
local_ip_for_outbound_sockets | 127.20.240.67 | experimental | 127.20.240.67:41087
local_ip_for_outbound_sockets | 127.20.240.68 | experimental | 127.20.240.68:44039
local_ip_for_outbound_sockets | 127.20.240.69 | experimental | 127.20.240.69:38795
local_ip_for_outbound_sockets | 127.20.240.70 | experimental | 127.20.240.70:34091
local_ip_for_outbound_sockets | 127.20.240.71 | experimental | 127.20.240.71:43179
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.240.65:36725
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.240.66:36169
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.240.67:41087
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.240.68:44039
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.240.69:38795
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.240.70:34091
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.240.71:43179
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.240.84:40571 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:40.644573 21441 log_verifier.cc:126] Checking tablet 1045468e57c1484995277f9ad5fc20b1
I20240503 14:12:40.659271 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1045468e57c1484995277f9ad5fc20b1
I20240503 14:12:40.659440 21441 log_verifier.cc:126] Checking tablet 1e3bf5d7cae7411486038ac34ab035e4
I20240503 14:12:40.671234 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1e3bf5d7cae7411486038ac34ab035e4
I20240503 14:12:40.671324 21441 log_verifier.cc:126] Checking tablet 2168c4283b0b4d63b6bb05fd20a41c9f
I20240503 14:12:40.683187 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2168c4283b0b4d63b6bb05fd20a41c9f
I20240503 14:12:40.683287 21441 log_verifier.cc:126] Checking tablet 3aba18dc398d4c8eb1dfce74850a1478
I20240503 14:12:40.695554 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3aba18dc398d4c8eb1dfce74850a1478
I20240503 14:12:40.695649 21441 log_verifier.cc:126] Checking tablet 4580e6da50584b93a7eab07537fa8a73
I20240503 14:12:40.707316 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4580e6da50584b93a7eab07537fa8a73
I20240503 14:12:40.707427 21441 log_verifier.cc:126] Checking tablet 4b33bbffc8d1476e91ac19f35c820305
I20240503 14:12:40.719146 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4b33bbffc8d1476e91ac19f35c820305
I20240503 14:12:40.719239 21441 log_verifier.cc:126] Checking tablet 54ff6aaa60734f548ff2b8da41ea5b3a
I20240503 14:12:40.731281 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 54ff6aaa60734f548ff2b8da41ea5b3a
I20240503 14:12:40.731388 21441 log_verifier.cc:126] Checking tablet 60a2f6744aa741d9b2af7eeefe868184
I20240503 14:12:40.743759 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 60a2f6744aa741d9b2af7eeefe868184
I20240503 14:12:40.743854 21441 log_verifier.cc:126] Checking tablet 60aad763f40242f98297320e9e8cf9ad
I20240503 14:12:40.755179 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 60aad763f40242f98297320e9e8cf9ad
I20240503 14:12:40.755271 21441 log_verifier.cc:126] Checking tablet 76c35354470b4d2187c8dd6c0d1594f4
I20240503 14:12:40.767611 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 76c35354470b4d2187c8dd6c0d1594f4
I20240503 14:12:40.767712 21441 log_verifier.cc:126] Checking tablet a71476f51de74a25ba84fc92ef0f9414
I20240503 14:12:40.779592 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a71476f51de74a25ba84fc92ef0f9414
I20240503 14:12:40.779692 21441 log_verifier.cc:126] Checking tablet d433f9489b204c8f9e1ef5c2164c5a6e
I20240503 14:12:40.791515 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d433f9489b204c8f9e1ef5c2164c5a6e
I20240503 14:12:40.791628 21441 log_verifier.cc:126] Checking tablet d7e3d00a314845b1959d4538b4c0a05e
I20240503 14:12:40.803756 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d7e3d00a314845b1959d4538b4c0a05e
I20240503 14:12:40.803860 21441 log_verifier.cc:126] Checking tablet db65a7aa1497429cb05bcd6b94d88914
I20240503 14:12:40.811394 21441 log_verifier.cc:177] Verified matching terms for 2 ops in tablet db65a7aa1497429cb05bcd6b94d88914
I20240503 14:12:40.811513 21441 log_verifier.cc:126] Checking tablet ed49476f299e4e8793bdfee31165ec8b
I20240503 14:12:40.824321 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ed49476f299e4e8793bdfee31165ec8b
I20240503 14:12:40.824429 21441 log_verifier.cc:126] Checking tablet feacaf3f7e794a19b20a4dea58730842
I20240503 14:12:40.836989 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet feacaf3f7e794a19b20a4dea58730842
I20240503 14:12:40.839983 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.72:0
--local_ip_for_outbound_sockets=127.20.240.72
--webserver_interface=127.20.240.72
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--builtin_ntp_servers=127.20.240.84:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:12:41.169368 27155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:41.169646 27155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:41.169778 27155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:41.177739 27155 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:41.177872 27155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:41.178023 27155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.72
I20240503 14:12:41.187332 27155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.240.72
--webserver_port=0
--tserver_master_addrs=127.20.240.126:43899
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.72
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:41.188654 27155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:41.190461 27155 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:41.203244 27163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:41.204035 27160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:41.204695 27161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:41.205456 27155 server_base.cc:1034] running on GCE node
I20240503 14:12:41.206003 27155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:41.206928 27155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:41.208117 27155 hybrid_clock.cc:648] HybridClock initialized: now 1714745561208062 us; error 55 us; skew 500 ppm
I20240503 14:12:41.208397 27155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:41.210538 27155 webserver.cc:460] Webserver started at http://127.20.240.72:41941/ using document root <none> and password file <none>
I20240503 14:12:41.211217 27155 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:41.211329 27155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:41.211620 27155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:41.215811 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4931bdb8349b4b478c39b8e7551abc12"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-xhmt"
I20240503 14:12:41.216524 27155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4931bdb8349b4b478c39b8e7551abc12"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-xhmt"
I20240503 14:12:41.221172 27155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:12:41.224334 27169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:41.225605 27155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:41.225785 27155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4931bdb8349b4b478c39b8e7551abc12"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-xhmt"
I20240503 14:12:41.225973 27155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:41.252848 27155 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:41.253652 27155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:41.253938 27155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:41.254951 27155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:41.256779 27155 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:41.256863 27155 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:41.256938 27155 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:41.256989 27155 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:41.290652 27155 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.72:42143
I20240503 14:12:41.290701 27281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.72:42143 every 8 connection(s)
I20240503 14:12:41.291914 27155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:12:41.298782 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 27155
I20240503 14:12:41.299023 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:12:41.305807 27282 heartbeater.cc:347] Connected to a master server at 127.20.240.126:43899
I20240503 14:12:41.306135 27282 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:41.307018 27282 heartbeater.cc:511] Master 127.20.240.126:43899 requested a full tablet report, sending...
I20240503 14:12:41.308204 23321 ts_manager.cc:194] Registered new tserver with Master: 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72:42143)
I20240503 14:12:41.308974 23321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.72:37129
I20240503 14:12:41.766424 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:41.766453 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:41.777000 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:41.798530 27217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:41.808523 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:41.850432 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:41.850847 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:41.874455 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:42.036100 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:42.048017 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:42.063760 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:42.071661 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:42.073348 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:42.074156 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:42.083214 27217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:42.114938 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:42.193969 23598 consensus_queue.cc:237] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
W20240503 14:12:42.202533 23532 consensus_peers.cc:485] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 -> Peer 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72:42143): Couldn't send request to peer 4931bdb8349b4b478c39b8e7551abc12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60aad763f40242f98297320e9e8cf9ad. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:42.311208 27282 heartbeater.cc:503] Master 127.20.240.126:43899 was elected leader, sending a full tablet report...
I20240503 14:12:42.708779 27389 ts_tablet_manager.cc:902] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: Initiating tablet copy from peer 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:42.710347 27389 tablet_copy_client.cc:308] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:36169
I20240503 14:12:42.716396 23618 tablet_copy_service.cc:140] P 7f3556100c8d46aaafb47e87defe8064: Received BeginTabletCopySession request for tablet 60aad763f40242f98297320e9e8cf9ad from peer 4931bdb8349b4b478c39b8e7551abc12 ({username='slave'} at 127.20.240.72:45869)
I20240503 14:12:42.716528 23618 tablet_copy_service.cc:161] P 7f3556100c8d46aaafb47e87defe8064: Beginning new tablet copy session on tablet 60aad763f40242f98297320e9e8cf9ad from peer 4931bdb8349b4b478c39b8e7551abc12 at {username='slave'} at 127.20.240.72:45869: session id = 4931bdb8349b4b478c39b8e7551abc12-60aad763f40242f98297320e9e8cf9ad
I20240503 14:12:42.718426 23618 tablet_copy_source_session.cc:205] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:42.722383 27389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60aad763f40242f98297320e9e8cf9ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:42.731343 27389 tablet_copy_client.cc:783] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:42.731586 27389 tablet_copy_client.cc:647] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:42.734010 27389 tablet_copy_client.cc:523] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:42.736846 27389 tablet_bootstrap.cc:492] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: Bootstrap starting.
I20240503 14:12:42.745239 27389 log.cc:824] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:42.774319 27389 tablet_bootstrap.cc:492] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:42.774832 27389 tablet_bootstrap.cc:492] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: Bootstrap complete.
I20240503 14:12:42.775195 27389 ts_tablet_manager.cc:1367] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: Time spent bootstrapping tablet: real 0.039s user 0.025s sys 0.015s
I20240503 14:12:42.779881 27389 raft_consensus.cc:348] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } }
I20240503 14:12:42.780658 27389 raft_consensus.cc:729] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4931bdb8349b4b478c39b8e7551abc12, State: Initialized, Role: FOLLOWER
I20240503 14:12:42.781020 27389 consensus_queue.cc:260] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:42.782475 27389 ts_tablet_manager.cc:1398] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20240503 14:12:42.783370 23618 tablet_copy_service.cc:342] P 7f3556100c8d46aaafb47e87defe8064: Request end of tablet copy session 4931bdb8349b4b478c39b8e7551abc12-60aad763f40242f98297320e9e8cf9ad received from {username='slave'} at 127.20.240.72:45869
I20240503 14:12:42.783496 23618 tablet_copy_service.cc:433] P 7f3556100c8d46aaafb47e87defe8064: ending tablet copy session 4931bdb8349b4b478c39b8e7551abc12-60aad763f40242f98297320e9e8cf9ad on tablet 60aad763f40242f98297320e9e8cf9ad with peer 4931bdb8349b4b478c39b8e7551abc12
I20240503 14:12:43.264978 27237 raft_consensus.cc:1203] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:43.266598 27399 raft_consensus.cc:2932] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:43.267297 27237 raft_consensus.cc:2932] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:43.269577 27399 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:45622 (request call id 0) took 1076 ms. Trace:
I20240503 14:12:43.269793 27399 rpcz_store.cc:276] 0503 14:12:42.193481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:12:42.193550 (+ 69us) service_pool.cc:225] Handling call
0503 14:12:42.194593 (+ 1043us) log.cc:842] Serialized 260 byte log entry
0503 14:12:43.269526 (+1074933us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":85,"raft.queue_time_us":370,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":309,"spinlock_wait_cycles":121216,"thread_start_us":458,"threads_started":2,"wal-append.queue_time_us":284,"wal-append.run_cpu_time_us":512,"wal-append.run_wall_time_us":1000581}
I20240503 14:12:43.272753 23323 catalog_manager.cc:5484] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 reported cstate change: config changed from index -1 to 4, VOTER 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "7f3556100c8d46aaafb47e87defe8064" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:43.302515 23466 consensus_queue.cc:237] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
W20240503 14:12:43.309900 23401 consensus_peers.cc:485] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 -> Peer 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72:42143): Couldn't send request to peer 4931bdb8349b4b478c39b8e7551abc12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b33bbffc8d1476e91ac19f35c820305. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:43.812021 27428 ts_tablet_manager.cc:902] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: Initiating tablet copy from peer 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:43.813086 27428 tablet_copy_client.cc:308] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:36725
I20240503 14:12:43.818703 23487 tablet_copy_service.cc:140] P 5ee9b8f29d3349f4ba05425d5287ff58: Received BeginTabletCopySession request for tablet 4b33bbffc8d1476e91ac19f35c820305 from peer 4931bdb8349b4b478c39b8e7551abc12 ({username='slave'} at 127.20.240.72:41801)
I20240503 14:12:43.818874 23487 tablet_copy_service.cc:161] P 5ee9b8f29d3349f4ba05425d5287ff58: Beginning new tablet copy session on tablet 4b33bbffc8d1476e91ac19f35c820305 from peer 4931bdb8349b4b478c39b8e7551abc12 at {username='slave'} at 127.20.240.72:41801: session id = 4931bdb8349b4b478c39b8e7551abc12-4b33bbffc8d1476e91ac19f35c820305
I20240503 14:12:43.820794 23487 tablet_copy_source_session.cc:205] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:43.821915 27428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b33bbffc8d1476e91ac19f35c820305. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:43.826706 27428 tablet_copy_client.cc:783] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:43.826962 27428 tablet_copy_client.cc:647] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:43.828498 27428 tablet_copy_client.cc:523] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:43.830847 27428 tablet_bootstrap.cc:492] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: Bootstrap starting.
I20240503 14:12:43.861213 27428 tablet_bootstrap.cc:492] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240503 14:12:43.861537 27428 tablet_bootstrap.cc:492] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: Bootstrap complete.
I20240503 14:12:43.861744 27428 ts_tablet_manager.cc:1367] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: Time spent bootstrapping tablet: real 0.031s user 0.026s sys 0.002s
I20240503 14:12:43.862536 27428 raft_consensus.cc:348] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } }
I20240503 14:12:43.862886 27428 raft_consensus.cc:729] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4931bdb8349b4b478c39b8e7551abc12, State: Initialized, Role: FOLLOWER
I20240503 14:12:43.863034 27428 consensus_queue.cc:260] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:43.863730 27428 ts_tablet_manager.cc:1398] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240503 14:12:43.864547 23487 tablet_copy_service.cc:342] P 5ee9b8f29d3349f4ba05425d5287ff58: Request end of tablet copy session 4931bdb8349b4b478c39b8e7551abc12-4b33bbffc8d1476e91ac19f35c820305 received from {username='slave'} at 127.20.240.72:41801
I20240503 14:12:43.864682 23487 tablet_copy_service.cc:433] P 5ee9b8f29d3349f4ba05425d5287ff58: ending tablet copy session 4931bdb8349b4b478c39b8e7551abc12-4b33bbffc8d1476e91ac19f35c820305 on tablet 4b33bbffc8d1476e91ac19f35c820305 with peer 4931bdb8349b4b478c39b8e7551abc12
I20240503 14:12:44.230434 27237 raft_consensus.cc:1203] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:44.231788 27438 raft_consensus.cc:2932] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:44.232439 27237 raft_consensus.cc:2932] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:44.236694 23317 catalog_manager.cc:5484] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 reported cstate change: config changed from index -1 to 4, VOTER 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:44.344748 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.345067 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.356518 27217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:44.371872 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.384759 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:44.405831 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.426154 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.428809 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:44.484385 23466 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b33bbffc8d1476e91ac19f35c820305"
dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49398
I20240503 14:12:44.484611 23466 raft_consensus.cc:595] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:44.576987 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.592123 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:44.594693 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.595942 23447 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:44.618386 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.628784 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.640717 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.647094 27217 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:44.663128 27438 raft_consensus.cc:979] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: : Instructing follower 4931bdb8349b4b478c39b8e7551abc12 to start an election
I20240503 14:12:44.663282 27438 raft_consensus.cc:1067] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Signalling peer 4931bdb8349b4b478c39b8e7551abc12 to start an election
I20240503 14:12:44.664166 27237 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b33bbffc8d1476e91ac19f35c820305"
dest_uuid: "4931bdb8349b4b478c39b8e7551abc12"
from {username='slave'} at 127.20.240.65:39773
I20240503 14:12:44.664444 27237 raft_consensus.cc:482] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:44.664615 27237 raft_consensus.cc:3037] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:44.667387 27237 raft_consensus.cc:504] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:44.675279 27237 leader_election.cc:288] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [CANDIDATE]: Term 2 election: Requested vote from peers 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725)
I20240503 14:12:44.676196 23466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b33bbffc8d1476e91ac19f35c820305" candidate_uuid: "4931bdb8349b4b478c39b8e7551abc12" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5ee9b8f29d3349f4ba05425d5287ff58"
I20240503 14:12:44.676416 23466 raft_consensus.cc:3032] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:44.676486 23466 raft_consensus.cc:729] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee9b8f29d3349f4ba05425d5287ff58, State: Running, Role: LEADER
I20240503 14:12:44.676733 23466 consensus_queue.cc:260] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:44.677029 23466 raft_consensus.cc:3037] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:44.680405 23466 raft_consensus.cc:2454] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4931bdb8349b4b478c39b8e7551abc12 in term 2.
I20240503 14:12:44.681150 27170 leader_election.cc:302] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4931bdb8349b4b478c39b8e7551abc12, 5ee9b8f29d3349f4ba05425d5287ff58; no voters:
I20240503 14:12:44.681703 27437 raft_consensus.cc:2781] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:44.691637 27437 raft_consensus.cc:686] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 2 LEADER]: Becoming Leader. State: Replica: 4931bdb8349b4b478c39b8e7551abc12, State: Running, Role: LEADER
I20240503 14:12:44.692201 27437 consensus_queue.cc:237] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:44.697165 23321 catalog_manager.cc:5484] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 reported cstate change: term changed from 1 to 2, leader changed from 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) to 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72). New cstate: current_term: 2 leader_uuid: "4931bdb8349b4b478c39b8e7551abc12" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:44.838047 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.853868 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.867869 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:44.869836 27217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:44.869876 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.869956 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.875511 23447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.901783 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:44.977087 23598 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "60aad763f40242f98297320e9e8cf9ad"
dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45672
I20240503 14:12:44.977329 23598 raft_consensus.cc:595] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:45.059638 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.074093 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.109840 23578 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:45.112973 23447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.113582 27217 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:45.124603 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.137696 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.137848 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.204511 23466 raft_consensus.cc:1261] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Refusing update from remote peer 4931bdb8349b4b478c39b8e7551abc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:45.205189 27614 consensus_queue.cc:1035] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee9b8f29d3349f4ba05425d5287ff58" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 36725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:45.398387 27620 raft_consensus.cc:979] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: : Instructing follower 4931bdb8349b4b478c39b8e7551abc12 to start an election
I20240503 14:12:45.398587 27619 raft_consensus.cc:1067] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Signalling peer 4931bdb8349b4b478c39b8e7551abc12 to start an election
I20240503 14:12:45.399240 27237 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "60aad763f40242f98297320e9e8cf9ad"
dest_uuid: "4931bdb8349b4b478c39b8e7551abc12"
from {username='slave'} at 127.20.240.66:48179
I20240503 14:12:45.399475 27237 raft_consensus.cc:482] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:45.399580 27237 raft_consensus.cc:3037] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:45.401675 27237 raft_consensus.cc:504] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:45.402289 27237 leader_election.cc:288] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [CANDIDATE]: Term 2 election: Requested vote from peers 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169)
I20240503 14:12:45.402799 23598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60aad763f40242f98297320e9e8cf9ad" candidate_uuid: "4931bdb8349b4b478c39b8e7551abc12" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7f3556100c8d46aaafb47e87defe8064"
I20240503 14:12:45.403012 23598 raft_consensus.cc:3032] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:45.403131 23598 raft_consensus.cc:729] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7f3556100c8d46aaafb47e87defe8064, State: Running, Role: LEADER
I20240503 14:12:45.403393 23598 consensus_queue.cc:260] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:45.403650 23598 raft_consensus.cc:3037] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:45.405583 23598 raft_consensus.cc:2454] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4931bdb8349b4b478c39b8e7551abc12 in term 2.
I20240503 14:12:45.406023 27173 leader_election.cc:302] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4931bdb8349b4b478c39b8e7551abc12, 7f3556100c8d46aaafb47e87defe8064; no voters:
I20240503 14:12:45.406333 27616 raft_consensus.cc:2781] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:45.406477 27616 raft_consensus.cc:686] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 2 LEADER]: Becoming Leader. State: Replica: 4931bdb8349b4b478c39b8e7551abc12, State: Running, Role: LEADER
I20240503 14:12:45.406741 27616 consensus_queue.cc:237] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:45.409130 23317 catalog_manager.cc:5484] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 reported cstate change: term changed from 1 to 2, leader changed from 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) to 4931bdb8349b4b478c39b8e7551abc12 (127.20.240.72). New cstate: current_term: 2 leader_uuid: "4931bdb8349b4b478c39b8e7551abc12" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 } } peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:45.504925 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.512336 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.514060 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.515908 27217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.526665 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.561125 23447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.584694 23578 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.589496 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.685832 27237 consensus_queue.cc:237] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:45.686638 27616 raft_consensus.cc:2932] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:45.691663 23323 catalog_manager.cc:5484] T 4b33bbffc8d1476e91ac19f35c820305 P 4931bdb8349b4b478c39b8e7551abc12 reported cstate change: config changed from index 4 to 6, VOTER 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "4931bdb8349b4b478c39b8e7551abc12" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:45.695374 23447 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b33bbffc8d1476e91ac19f35c820305 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee9b8f29d3349f4ba05425d5287ff58 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53126
I20240503 14:12:45.696084 27681 tablet_replica.cc:330] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: stopping tablet replica
I20240503 14:12:45.696277 27681 raft_consensus.cc:2229] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:45.696480 27681 raft_consensus.cc:2258] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:45.698215 27681 ts_tablet_manager.cc:1875] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:45.703078 27681 ts_tablet_manager.cc:1888] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:45.703213 27681 log.cc:1196] T 4b33bbffc8d1476e91ac19f35c820305 P 5ee9b8f29d3349f4ba05425d5287ff58: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/4b33bbffc8d1476e91ac19f35c820305
I20240503 14:12:45.703925 23310 catalog_manager.cc:4830] TS 5ee9b8f29d3349f4ba05425d5287ff58 (127.20.240.65:36725): tablet 4b33bbffc8d1476e91ac19f35c820305 (table rebalance_test_table_2 [id=13c93033066d40228c6279535bf62553]) successfully deleted
I20240503 14:12:45.784262 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.787703 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.788586 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.802942 27217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.804603 23447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.810181 23598 raft_consensus.cc:1261] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Refusing update from remote peer 4931bdb8349b4b478c39b8e7551abc12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:12:45.810809 27616 consensus_queue.cc:1035] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f3556100c8d46aaafb47e87defe8064" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:45.823258 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.846221 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.877121 23578 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:45.980247 27237 consensus_queue.cc:237] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } }
I20240503 14:12:45.981209 27616 raft_consensus.cc:2932] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:45.984969 23326 catalog_manager.cc:5484] T 60aad763f40242f98297320e9e8cf9ad P 4931bdb8349b4b478c39b8e7551abc12 reported cstate change: config changed from index 4 to 6, VOTER 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66) evicted. New cstate: current_term: 2 leader_uuid: "4931bdb8349b4b478c39b8e7551abc12" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4931bdb8349b4b478c39b8e7551abc12" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 42143 } } }
I20240503 14:12:45.988988 23578 tablet_service.cc:1514] Processing DeleteTablet for tablet 60aad763f40242f98297320e9e8cf9ad with delete_type TABLET_DATA_TOMBSTONED (TS 7f3556100c8d46aaafb47e87defe8064 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37200
I20240503 14:12:45.989701 27733 tablet_replica.cc:330] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: stopping tablet replica
I20240503 14:12:45.989938 27733 raft_consensus.cc:2229] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:45.990190 27733 raft_consensus.cc:2258] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:45.992278 27733 ts_tablet_manager.cc:1875] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:46.000241 27733 ts_tablet_manager.cc:1888] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:12:46.000394 27733 log.cc:1196] T 60aad763f40242f98297320e9e8cf9ad P 7f3556100c8d46aaafb47e87defe8064: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/60aad763f40242f98297320e9e8cf9ad
I20240503 14:12:46.001289 23313 catalog_manager.cc:4830] TS 7f3556100c8d46aaafb47e87defe8064 (127.20.240.66:36169): tablet 60aad763f40242f98297320e9e8cf9ad (table rebalance_test_table_3 [id=14847a692cab4bf487630962c7ba1934]) successfully deleted
I20240503 14:12:46.307528 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.319808 27217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.329213 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.346058 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.350070 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.367868 23578 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.387090 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.410053 23447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.556419 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.558787 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.565043 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.572316 27217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.588924 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.591018 23578 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.611572 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.643218 23447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.927522 25209 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.937736 23447 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.943149 24677 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.960160 24809 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:46.968731 24949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:47.001857 23578 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:47.002728 27217 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:47.019228 25079 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4c8d95e4965f4cb2bfdbc452fe5f7e0b | 127.20.240.126:43899 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.240.84:40571 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
0e71aaeb321d4364bc1d81d64815b201 | 127.20.240.71:43179 | HEALTHY | <none> | 2 | 0
1a4466679fbc43618c8fb2e3d8a26dfd | 127.20.240.67:41087 | HEALTHY | <none> | 2 | 0
2062470ed9164d389aa2e60973ecdcdb | 127.20.240.68:44039 | HEALTHY | <none> | 2 | 0
4931bdb8349b4b478c39b8e7551abc12 | 127.20.240.72:42143 | HEALTHY | <none> | 2 | 0
4fbbe1e5ec864182b3116ef1278f7a3a | 127.20.240.69:38795 | HEALTHY | <none> | 2 | 0
5ee9b8f29d3349f4ba05425d5287ff58 | 127.20.240.65:36725 | HEALTHY | <none> | 2 | 0
7f3556100c8d46aaafb47e87defe8064 | 127.20.240.66:36169 | HEALTHY | <none> | 2 | 0
95dfe9d1f79f4a8696fc5586e3ea3731 | 127.20.240.70:34091 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.20.240.65 | experimental | 127.20.240.65:36725
local_ip_for_outbound_sockets | 127.20.240.66 | experimental | 127.20.240.66:36169
local_ip_for_outbound_sockets | 127.20.240.67 | experimental | 127.20.240.67:41087
local_ip_for_outbound_sockets | 127.20.240.68 | experimental | 127.20.240.68:44039
local_ip_for_outbound_sockets | 127.20.240.69 | experimental | 127.20.240.69:38795
local_ip_for_outbound_sockets | 127.20.240.70 | experimental | 127.20.240.70:34091
local_ip_for_outbound_sockets | 127.20.240.71 | experimental | 127.20.240.71:43179
local_ip_for_outbound_sockets | 127.20.240.72 | experimental | 127.20.240.72:42143
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.240.65:36725
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.240.66:36169
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.240.67:41087
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.20.240.68:44039
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.20.240.69:38795
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.20.240.70:34091
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.20.240.71:43179
server_dump_info_path | /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.20.240.72:42143
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.240.84:40571 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:47.088167 21441 log_verifier.cc:126] Checking tablet 1045468e57c1484995277f9ad5fc20b1
I20240503 14:12:47.097501 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1045468e57c1484995277f9ad5fc20b1
I20240503 14:12:47.097589 21441 log_verifier.cc:126] Checking tablet 1e3bf5d7cae7411486038ac34ab035e4
I20240503 14:12:47.106735 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1e3bf5d7cae7411486038ac34ab035e4
I20240503 14:12:47.106820 21441 log_verifier.cc:126] Checking tablet 2168c4283b0b4d63b6bb05fd20a41c9f
I20240503 14:12:47.115563 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2168c4283b0b4d63b6bb05fd20a41c9f
I20240503 14:12:47.115680 21441 log_verifier.cc:126] Checking tablet 3aba18dc398d4c8eb1dfce74850a1478
I20240503 14:12:47.125133 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3aba18dc398d4c8eb1dfce74850a1478
I20240503 14:12:47.125217 21441 log_verifier.cc:126] Checking tablet 4580e6da50584b93a7eab07537fa8a73
I20240503 14:12:47.134430 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4580e6da50584b93a7eab07537fa8a73
I20240503 14:12:47.134516 21441 log_verifier.cc:126] Checking tablet 4b33bbffc8d1476e91ac19f35c820305
I20240503 14:12:47.146876 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4b33bbffc8d1476e91ac19f35c820305
I20240503 14:12:47.146979 21441 log_verifier.cc:126] Checking tablet 54ff6aaa60734f548ff2b8da41ea5b3a
I20240503 14:12:47.156479 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 54ff6aaa60734f548ff2b8da41ea5b3a
I20240503 14:12:47.156566 21441 log_verifier.cc:126] Checking tablet 60a2f6744aa741d9b2af7eeefe868184
I20240503 14:12:47.166371 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 60a2f6744aa741d9b2af7eeefe868184
I20240503 14:12:47.166458 21441 log_verifier.cc:126] Checking tablet 60aad763f40242f98297320e9e8cf9ad
I20240503 14:12:47.178524 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 60aad763f40242f98297320e9e8cf9ad
I20240503 14:12:47.178622 21441 log_verifier.cc:126] Checking tablet 76c35354470b4d2187c8dd6c0d1594f4
I20240503 14:12:47.187927 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 76c35354470b4d2187c8dd6c0d1594f4
I20240503 14:12:47.188011 21441 log_verifier.cc:126] Checking tablet a71476f51de74a25ba84fc92ef0f9414
I20240503 14:12:47.197271 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a71476f51de74a25ba84fc92ef0f9414
I20240503 14:12:47.197360 21441 log_verifier.cc:126] Checking tablet d433f9489b204c8f9e1ef5c2164c5a6e
I20240503 14:12:47.206416 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d433f9489b204c8f9e1ef5c2164c5a6e
I20240503 14:12:47.206497 21441 log_verifier.cc:126] Checking tablet d7e3d00a314845b1959d4538b4c0a05e
I20240503 14:12:47.215963 21441 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d7e3d00a314845b1959d4538b4c0a05e
I20240503 14:12:47.216039 21441 log_verifier.cc:126] Checking tablet db65a7aa1497429cb05bcd6b94d88914
I20240503 14:12:47.220685 21441 log_verifier.cc:177] Verified matching terms for 2 ops in tablet db65a7aa1497429cb05bcd6b94d88914
I20240503 14:12:47.220757 21441 log_verifier.cc:126] Checking tablet ed49476f299e4e8793bdfee31165ec8b
I20240503 14:12:47.229869 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ed49476f299e4e8793bdfee31165ec8b
I20240503 14:12:47.229946 21441 log_verifier.cc:126] Checking tablet feacaf3f7e794a19b20a4dea58730842
I20240503 14:12:47.239275 21441 log_verifier.cc:177] Verified matching terms for 6 ops in tablet feacaf3f7e794a19b20a4dea58730842
I20240503 14:12:47.240072 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 23385
I20240503 14:12:47.241359 23506 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:47.445973 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 23385
I20240503 14:12:47.478621 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 23516
I20240503 14:12:47.479609 23638 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:47.680246 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 23516
I20240503 14:12:47.712936 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 24615
I20240503 14:12:47.718669 24737 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:47.891678 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 24615
I20240503 14:12:47.919849 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 24746
I20240503 14:12:47.926012 24870 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:48.098850 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 24746
I20240503 14:12:48.127416 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 24881
I20240503 14:12:48.134063 25008 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:48.306563 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 24881
I20240503 14:12:48.333900 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 25017
I20240503 14:12:48.339630 25139 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:48.515995 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 25017
I20240503 14:12:48.543687 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 25147
I20240503 14:12:48.550460 25269 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:48.724318 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 25147
I20240503 14:12:48.752180 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 27155
I20240503 14:12:48.757805 27277 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:48.868364 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 27155
I20240503 14:12:48.887883 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 23295
I20240503 14:12:48.888891 23356 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:49.111575 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 23295
2024-05-03T14:12:49Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (43017 ms)
[----------] 1 test from RebalanceParamTest (43017 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240503 14:12:49.189169 21441 test_util.cc:268] Using random seed: 1351097625
I20240503 14:12:49.191027 21441 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:49.191088 21441 ts_itest-base.cc:113] --------------
I20240503 14:12:49.191138 21441 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:12:49.191184 21441 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:12:49.191246 21441 ts_itest-base.cc:116] --------------
2024-05-03T14:12:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:49Z Disabled control of system clock
I20240503 14:12:49.218830 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:33033
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:33033
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 14:12:49.529667 27871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:49.529968 27871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:49.530108 27871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:49.537242 27871 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:49.537344 27871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:49.537429 27871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:49.537513 27871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:49.537585 27871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:49.546021 27871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:33033
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:33033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:49.547222 27871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:49.548905 27871 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:49.559965 27876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:49.560657 27877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:49.561540 27879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:49.562446 27871 server_base.cc:1034] running on GCE node
I20240503 14:12:49.563153 27871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:49.564483 27871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:49.565683 27871 hybrid_clock.cc:648] HybridClock initialized: now 1714745569565638 us; error 47 us; skew 500 ppm
I20240503 14:12:49.565981 27871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:49.568113 27871 webserver.cc:460] Webserver started at http://127.20.240.126:33939/ using document root <none> and password file <none>
I20240503 14:12:49.568729 27871 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:49.568828 27871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:49.569097 27871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:49.571579 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5baa17487b4b4e1eb60d2f5fcb4de355"
format_stamp: "Formatted at 2024-05-03 14:12:49 on dist-test-slave-xhmt"
I20240503 14:12:49.572275 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5baa17487b4b4e1eb60d2f5fcb4de355"
format_stamp: "Formatted at 2024-05-03 14:12:49 on dist-test-slave-xhmt"
I20240503 14:12:49.576880 27871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240503 14:12:49.579790 27885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:49.580915 27871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:49.581065 27871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5baa17487b4b4e1eb60d2f5fcb4de355"
format_stamp: "Formatted at 2024-05-03 14:12:49 on dist-test-slave-xhmt"
I20240503 14:12:49.581255 27871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:49.608414 27871 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:49.609171 27871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:49.609447 27871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:49.625937 27871 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:33033
I20240503 14:12:49.625977 27936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:33033 every 8 connection(s)
I20240503 14:12:49.627194 27871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:49.630640 27937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.636027 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 27871
I20240503 14:12:49.636243 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:49.640305 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Bootstrap starting.
I20240503 14:12:49.643443 27937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.644438 27937 log.cc:824] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:49.647080 27937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: No bootstrap required, opened a new log
I20240503 14:12:49.653002 27937 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33033 } }
I20240503 14:12:49.653306 27937 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.653406 27937 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5baa17487b4b4e1eb60d2f5fcb4de355, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.653980 27937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33033 } }
I20240503 14:12:49.654176 27937 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:49.654309 27937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:49.654448 27937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:49.656394 27937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33033 } }
I20240503 14:12:49.656807 27937 leader_election.cc:302] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5baa17487b4b4e1eb60d2f5fcb4de355; no voters:
I20240503 14:12:49.657300 27937 leader_election.cc:288] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:49.657465 27942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:49.658082 27942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [term 1 LEADER]: Becoming Leader. State: Replica: 5baa17487b4b4e1eb60d2f5fcb4de355, State: Running, Role: LEADER
I20240503 14:12:49.658676 27942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33033 } }
I20240503 14:12:49.659569 27937 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:49.661271 27943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5baa17487b4b4e1eb60d2f5fcb4de355. Latest consensus state: current_term: 1 leader_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33033 } } }
I20240503 14:12:49.661650 27943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:49.662581 27944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5baa17487b4b4e1eb60d2f5fcb4de355" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 33033 } } }
I20240503 14:12:49.663334 27944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:49.664250 27950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:49.669273 27950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:49.676963 27950 catalog_manager.cc:1347] Generated new cluster ID: 136cb16d62574c4685f08a0fa45c4f7a
I20240503 14:12:49.677060 27950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:49.696277 27950 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:49.697368 27950 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:49.705871 27950 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Generated new TSK 0
I20240503 14:12:49.706604 27950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:49.717304 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:12:50.032862 27961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:50.033149 27961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:50.033242 27961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:50.040431 27961 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:50.040534 27961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:50.040686 27961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20240503 14:12:50.049026 27961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:50.050216 27961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:50.051872 27961 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:50.063197 27966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:50.064078 27967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:50.064704 27969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:50.065516 27961 server_base.cc:1034] running on GCE node
I20240503 14:12:50.066082 27961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:50.066975 27961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:50.068197 27961 hybrid_clock.cc:648] HybridClock initialized: now 1714745570068156 us; error 26 us; skew 500 ppm
I20240503 14:12:50.068526 27961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:50.070724 27961 webserver.cc:460] Webserver started at http://127.20.240.65:44045/ using document root <none> and password file <none>
I20240503 14:12:50.071422 27961 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:50.071528 27961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:50.071817 27961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:50.074276 27961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.075031 27961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.079762 27961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:50.082823 27975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.083962 27961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240503 14:12:50.084121 27961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.084293 27961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:50.114076 27961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:50.114903 27961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:50.115175 27961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:50.116079 27961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:50.117498 27961 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:50.117597 27961 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.117713 27961 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:50.117790 27961 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.147552 27961 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:33293
I20240503 14:12:50.147622 28087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:33293 every 8 connection(s)
I20240503 14:12:50.148993 27961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:50.158155 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 27961
I20240503 14:12:50.158417 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:50.160674 28088 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:50.160979 28088 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:50.161818 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:50.161859 28088 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:50.163825 27902 ts_manager.cc:194] Registered new tserver with Master: 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:12:50.165447 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:45983
W20240503 14:12:50.481465 28092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:50.481786 28092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:50.481920 28092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:50.489190 28092 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:50.489320 28092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:50.489468 28092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20240503 14:12:50.498284 28092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:50.499586 28092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:50.501269 28092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:50.512926 28098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:50.513720 28097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:50.514415 28100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:50.515146 28092 server_base.cc:1034] running on GCE node
I20240503 14:12:50.515619 28092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:50.516466 28092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:50.517649 28092 hybrid_clock.cc:648] HybridClock initialized: now 1714745570517610 us; error 33 us; skew 500 ppm
I20240503 14:12:50.517943 28092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:50.520020 28092 webserver.cc:460] Webserver started at http://127.20.240.66:37881/ using document root <none> and password file <none>
I20240503 14:12:50.520700 28092 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:50.520794 28092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:50.521063 28092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:50.523478 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6409961ee8c0445687c11724213c44a9"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.524194 28092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6409961ee8c0445687c11724213c44a9"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.528807 28092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:12:50.531795 28106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.532938 28092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:50.533078 28092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6409961ee8c0445687c11724213c44a9"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.533249 28092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:50.562178 28092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:50.562996 28092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:50.563253 28092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:50.564167 28092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:50.565629 28092 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:50.565723 28092 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.565831 28092 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:50.565913 28092 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.595964 28092 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:37039
I20240503 14:12:50.596047 28218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:37039 every 8 connection(s)
I20240503 14:12:50.597203 28092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:50.601538 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 28092
I20240503 14:12:50.601735 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:50.605134 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:50.610509 28219 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:50.610781 28219 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:50.611454 28219 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:50.612663 27902 ts_manager.cc:194] Registered new tserver with Master: 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:12:50.613396 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:40387
W20240503 14:12:50.920986 28222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:50.921268 28222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:50.921387 28222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:50.928332 28222 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:50.928460 28222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:50.928617 28222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20240503 14:12:50.937105 28222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:50.938369 28222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:50.939996 28222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:50.951381 28227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:50.952179 28228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:50.952859 28230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:50.953568 28222 server_base.cc:1034] running on GCE node
I20240503 14:12:50.954094 28222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:50.954974 28222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:50.956184 28222 hybrid_clock.cc:648] HybridClock initialized: now 1714745570956124 us; error 35 us; skew 500 ppm
I20240503 14:12:50.956470 28222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:50.958598 28222 webserver.cc:460] Webserver started at http://127.20.240.67:42993/ using document root <none> and password file <none>
I20240503 14:12:50.959275 28222 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:50.959381 28222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:50.959684 28222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:50.962098 28222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a4cdf05256a0407c8929ac43eb55d44f"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.962878 28222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a4cdf05256a0407c8929ac43eb55d44f"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.967672 28222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240503 14:12:50.970629 28236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.971872 28222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20240503 14:12:50.972011 28222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a4cdf05256a0407c8929ac43eb55d44f"
format_stamp: "Formatted at 2024-05-03 14:12:50 on dist-test-slave-xhmt"
I20240503 14:12:50.972179 28222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:50.993616 28222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:50.994349 28222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:50.994599 28222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:50.995433 28222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:50.996759 28222 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:50.996852 28222 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:50.996958 28222 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:50.997030 28222 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:51.026767 28222 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:43889
I20240503 14:12:51.026816 28348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:43889 every 8 connection(s)
I20240503 14:12:51.027951 28222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:51.036675 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 28222
I20240503 14:12:51.036845 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:51.040117 28349 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:51.040122 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.68:0
--local_ip_for_outbound_sockets=127.20.240.68
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:51.040382 28349 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:51.041035 28349 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:51.042333 27902 ts_manager.cc:194] Registered new tserver with Master: a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:12:51.043326 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:36425
I20240503 14:12:51.168125 28088 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
W20240503 14:12:51.355708 28352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:51.356076 28352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:51.356232 28352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:51.364256 28352 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:51.364388 28352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:51.364567 28352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.68
I20240503 14:12:51.373435 28352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.68
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:51.374770 28352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:51.376508 28352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:51.388550 28360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:51.389151 28357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:51.389719 28358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:51.390827 28352 server_base.cc:1034] running on GCE node
I20240503 14:12:51.391335 28352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:51.392225 28352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:51.393424 28352 hybrid_clock.cc:648] HybridClock initialized: now 1714745571393382 us; error 40 us; skew 500 ppm
I20240503 14:12:51.393740 28352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:51.400185 28352 webserver.cc:460] Webserver started at http://127.20.240.68:34267/ using document root <none> and password file <none>
I20240503 14:12:51.400857 28352 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:51.400972 28352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:51.401285 28352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:51.405349 28352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "5388902973134958a030fd63ab4faba4"
format_stamp: "Formatted at 2024-05-03 14:12:51 on dist-test-slave-xhmt"
I20240503 14:12:51.406131 28352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "5388902973134958a030fd63ab4faba4"
format_stamp: "Formatted at 2024-05-03 14:12:51 on dist-test-slave-xhmt"
I20240503 14:12:51.410794 28352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 14:12:51.413741 28366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:51.414938 28352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:51.415084 28352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "5388902973134958a030fd63ab4faba4"
format_stamp: "Formatted at 2024-05-03 14:12:51 on dist-test-slave-xhmt"
I20240503 14:12:51.415256 28352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:51.433746 28352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:51.434515 28352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:51.434762 28352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:51.435585 28352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:51.436940 28352 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:51.437044 28352 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:51.437152 28352 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:51.437224 28352 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:51.469457 28352 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.68:38371
I20240503 14:12:51.469532 28478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.68:38371 every 8 connection(s)
I20240503 14:12:51.470793 28352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:51.479103 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 28352
I20240503 14:12:51.479275 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:51.482620 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:0
--local_ip_for_outbound_sockets=127.20.240.69
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:51.483004 28479 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:51.483314 28479 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:51.484133 28479 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:51.485453 27902 ts_manager.cc:194] Registered new tserver with Master: 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:12:51.486261 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.68:56203
I20240503 14:12:51.615689 28219 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
W20240503 14:12:51.796798 28482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:51.797112 28482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:51.797248 28482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:51.804313 28482 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:51.804437 28482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:51.804589 28482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:12:51.812806 28482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:51.813979 28482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:51.815625 28482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:51.827504 28487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:51.828260 28488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:51.828987 28490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:51.829660 28482 server_base.cc:1034] running on GCE node
I20240503 14:12:51.830173 28482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:51.831310 28482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:51.832509 28482 hybrid_clock.cc:648] HybridClock initialized: now 1714745571832469 us; error 37 us; skew 500 ppm
I20240503 14:12:51.832784 28482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:51.834864 28482 webserver.cc:460] Webserver started at http://127.20.240.69:38643/ using document root <none> and password file <none>
I20240503 14:12:51.835539 28482 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:51.835649 28482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:51.835942 28482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:51.838476 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d0dbfb0e43f948c3922926aacc365fc1"
format_stamp: "Formatted at 2024-05-03 14:12:51 on dist-test-slave-xhmt"
I20240503 14:12:51.839231 28482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d0dbfb0e43f948c3922926aacc365fc1"
format_stamp: "Formatted at 2024-05-03 14:12:51 on dist-test-slave-xhmt"
I20240503 14:12:51.843813 28482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:12:51.847155 28496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:51.848361 28482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:51.848510 28482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d0dbfb0e43f948c3922926aacc365fc1"
format_stamp: "Formatted at 2024-05-03 14:12:51 on dist-test-slave-xhmt"
I20240503 14:12:51.848693 28482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:51.867736 28482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:51.868541 28482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:51.868784 28482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:51.869640 28482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:51.871090 28482 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:51.871174 28482 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:51.871254 28482 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:51.871307 28482 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:51.900569 28482 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:39689
I20240503 14:12:51.900622 28608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:39689 every 8 connection(s)
I20240503 14:12:51.901856 28482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:51.911609 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 28482
I20240503 14:12:51.911792 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:51.914166 28609 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:51.914489 28609 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:51.915007 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.70:0
--local_ip_for_outbound_sockets=127.20.240.70
--webserver_interface=127.20.240.70
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:51.915264 28609 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:51.916611 27902 ts_manager.cc:194] Registered new tserver with Master: d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:12:51.917393 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:55799
I20240503 14:12:52.045565 28349 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
W20240503 14:12:52.229640 28612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:52.229903 28612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:52.229992 28612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:52.237267 28612 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:52.237372 28612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:52.237529 28612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.70
I20240503 14:12:52.245998 28612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.240.70
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.70
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:52.247184 28612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:52.248795 28612 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:52.260711 28620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:52.261377 28618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:52.261889 28617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:52.262913 28612 server_base.cc:1034] running on GCE node
I20240503 14:12:52.263440 28612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:52.264344 28612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:52.265555 28612 hybrid_clock.cc:648] HybridClock initialized: now 1714745572265512 us; error 57 us; skew 500 ppm
I20240503 14:12:52.265861 28612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:52.267974 28612 webserver.cc:460] Webserver started at http://127.20.240.70:39505/ using document root <none> and password file <none>
I20240503 14:12:52.268620 28612 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:52.268711 28612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:52.268960 28612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:52.271415 28612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6bbc686598f24f518d7ca6bb2063058d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:12:52.272119 28612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6bbc686598f24f518d7ca6bb2063058d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:12:52.276906 28612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:52.279938 28626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:52.281121 28612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:52.281261 28612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6bbc686598f24f518d7ca6bb2063058d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:12:52.281432 28612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:52.300711 28612 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:52.301440 28612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:52.301680 28612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:52.302534 28612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:52.303833 28612 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:52.303921 28612 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:52.304033 28612 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:52.304103 28612 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:52.333138 28612 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.70:39465
I20240503 14:12:52.333187 28738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.70:39465 every 8 connection(s)
I20240503 14:12:52.334417 28612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:52.343623 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 28612
I20240503 14:12:52.343813 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:12:52.346287 28739 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:52.346612 28739 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:52.347062 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.71:0
--local_ip_for_outbound_sockets=127.20.240.71
--webserver_interface=127.20.240.71
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:52.347349 28739 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:52.348603 27902 ts_manager.cc:194] Registered new tserver with Master: 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:12:52.349421 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.70:35319
I20240503 14:12:52.488808 28479 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
W20240503 14:12:52.665406 28742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:52.665681 28742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:52.665782 28742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:52.672930 28742 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:52.673040 28742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:52.673151 28742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.71
I20240503 14:12:52.681820 28742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.240.71
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.71
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:52.683058 28742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:52.684657 28742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:52.696614 28747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:52.697185 28750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:52.698048 28748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:52.698879 28742 server_base.cc:1034] running on GCE node
I20240503 14:12:52.699388 28742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:52.700245 28742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:52.701431 28742 hybrid_clock.cc:648] HybridClock initialized: now 1714745572701389 us; error 37 us; skew 500 ppm
I20240503 14:12:52.701725 28742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:52.703827 28742 webserver.cc:460] Webserver started at http://127.20.240.71:39749/ using document root <none> and password file <none>
I20240503 14:12:52.704516 28742 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:52.704628 28742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:52.704936 28742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:52.707417 28742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "fde134d0fa2b4900b9badd2e08fea09d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:12:52.708174 28742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "fde134d0fa2b4900b9badd2e08fea09d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:12:52.713097 28742 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240503 14:12:52.716305 28756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:52.717577 28742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:52.717726 28742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fde134d0fa2b4900b9badd2e08fea09d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:12:52.717916 28742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:52.736136 28742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:52.736914 28742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:52.737157 28742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:52.738126 28742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:52.739542 28742 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:52.739637 28742 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:52.739748 28742 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:52.739821 28742 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:52.770570 28742 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.71:34707
I20240503 14:12:52.770645 28868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.71:34707 every 8 connection(s)
I20240503 14:12:52.771824 28742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:52.775908 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 28742
I20240503 14:12:52.776122 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:12:52.779692 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.72:0
--local_ip_for_outbound_sockets=127.20.240.72
--webserver_interface=127.20.240.72
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:12:52.785280 28869 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:52.785629 28869 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:52.786443 28869 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:52.787645 27902 ts_manager.cc:194] Registered new tserver with Master: fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:12:52.788452 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.71:37949
I20240503 14:12:52.919801 28609 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
W20240503 14:12:53.097211 28872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:53.097486 28872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:53.097626 28872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:53.104507 28872 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:12:53.104632 28872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:53.104779 28872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.72
I20240503 14:12:53.113314 28872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.240.72
--webserver_port=0
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.72
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:53.114585 28872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:53.116187 28872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:53.127499 28877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.128345 28878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:53.128989 28880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:53.129770 28872 server_base.cc:1034] running on GCE node
I20240503 14:12:53.130355 28872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:53.131217 28872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:53.132416 28872 hybrid_clock.cc:648] HybridClock initialized: now 1714745573132373 us; error 37 us; skew 500 ppm
I20240503 14:12:53.132709 28872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:53.141307 28872 webserver.cc:460] Webserver started at http://127.20.240.72:40733/ using document root <none> and password file <none>
I20240503 14:12:53.142063 28872 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:53.142174 28872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:53.142524 28872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:53.145108 28872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "530893a961af4b95b6b687fa5ed22f9d"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-xhmt"
I20240503 14:12:53.145828 28872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "530893a961af4b95b6b687fa5ed22f9d"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-xhmt"
I20240503 14:12:53.151679 28872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20240503 14:12:53.155018 28886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.156273 28872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:53.156424 28872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "530893a961af4b95b6b687fa5ed22f9d"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-xhmt"
I20240503 14:12:53.156603 28872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:53.173193 28872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:53.174028 28872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:53.174285 28872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:53.175124 28872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:53.176573 28872 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:53.176659 28872 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.176738 28872 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:53.176790 28872 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:53.215478 28872 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.72:41331
I20240503 14:12:53.215802 28998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.72:41331 every 8 connection(s)
I20240503 14:12:53.216969 28872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:12:53.220309 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 28872
I20240503 14:12:53.220507 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:12:53.231429 28999 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:12:53.231722 28999 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:53.232396 28999 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:12:53.233772 27902 ts_manager.cc:194] Registered new tserver with Master: 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:12:53.235008 27902 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.72:48047
I20240503 14:12:53.237632 21441 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:12:53.252602 21441 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:12:53.252724 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 28482
I20240503 14:12:53.260226 28604 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:53.339207 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 28482
I20240503 14:12:53.351680 28739 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
I20240503 14:12:53.355093 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 28612
I20240503 14:12:53.362926 28734 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:53.442342 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 28612
I20240503 14:12:53.457581 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 28742
I20240503 14:12:53.464998 28864 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:53.543378 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 28742
I20240503 14:12:53.559095 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 28872
I20240503 14:12:53.567610 28994 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:53.647090 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 28872
I20240503 14:12:57.414932 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:57.453212 28414 tablet_service.cc:1467] Processing CreateTablet for tablet c26fbb09f31c4c8f81788c784da6f205 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.454154 28413 tablet_service.cc:1467] Processing CreateTablet for tablet aa5031adcf1b4e55a73b615080fbd73c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.455205 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5031adcf1b4e55a73b615080fbd73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.459543 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26fbb09f31c4c8f81788c784da6f205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.460525 28154 tablet_service.cc:1467] Processing CreateTablet for tablet c26fbb09f31c4c8f81788c784da6f205 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.461061 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 5867eb31f7344b178db1a2d0a22da70b (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.462050 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5867eb31f7344b178db1a2d0a22da70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.462983 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26fbb09f31c4c8f81788c784da6f205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.464673 28021 tablet_service.cc:1467] Processing CreateTablet for tablet aa5031adcf1b4e55a73b615080fbd73c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.465005 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 5867eb31f7344b178db1a2d0a22da70b (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.465656 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 5867eb31f7344b178db1a2d0a22da70b (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.465879 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5031adcf1b4e55a73b615080fbd73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.466475 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5867eb31f7344b178db1a2d0a22da70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.466351 28283 tablet_service.cc:1467] Processing CreateTablet for tablet aa5031adcf1b4e55a73b615080fbd73c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.466248 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5867eb31f7344b178db1a2d0a22da70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.468339 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa5031adcf1b4e55a73b615080fbd73c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.464021 28023 tablet_service.cc:1467] Processing CreateTablet for tablet c26fbb09f31c4c8f81788c784da6f205 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.473100 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c26fbb09f31c4c8f81788c784da6f205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.485324 29029 tablet_bootstrap.cc:492] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:12:57.489580 29027 tablet_bootstrap.cc:492] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:12:57.491081 29029 tablet_bootstrap.cc:654] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.492352 29029 log.cc:824] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:57.493273 29027 tablet_bootstrap.cc:654] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.493525 29028 tablet_bootstrap.cc:492] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:12:57.494685 29027 log.cc:824] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:57.496927 29029 tablet_bootstrap.cc:492] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:12:57.496924 29028 tablet_bootstrap.cc:654] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.497669 29029 ts_tablet_manager.cc:1367] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20240503 14:12:57.498080 29028 log.cc:824] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:57.501247 29031 tablet_bootstrap.cc:492] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:12:57.503062 29027 tablet_bootstrap.cc:492] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:12:57.503315 29027 ts_tablet_manager.cc:1367] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 14:12:57.505307 29031 tablet_bootstrap.cc:654] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.506461 29031 log.cc:824] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:57.509447 29029 raft_consensus.cc:348] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.510113 29029 raft_consensus.cc:374] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.510313 29029 raft_consensus.cc:729] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.511368 29029 consensus_queue.cc:260] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.513971 29027 raft_consensus.cc:348] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.514439 29027 raft_consensus.cc:374] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.514569 29027 raft_consensus.cc:729] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.515234 29027 consensus_queue.cc:260] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.515497 29029 ts_tablet_manager.cc:1398] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20240503 14:12:57.515928 29029 tablet_bootstrap.cc:492] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:12:57.517419 29027 ts_tablet_manager.cc:1398] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20240503 14:12:57.517567 29029 tablet_bootstrap.cc:654] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.517828 29027 tablet_bootstrap.cc:492] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:12:57.519531 29027 tablet_bootstrap.cc:654] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.519711 29028 tablet_bootstrap.cc:492] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:12:57.520097 29028 ts_tablet_manager.cc:1367] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.004s
I20240503 14:12:57.524794 29031 tablet_bootstrap.cc:492] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:12:57.525928 29029 tablet_bootstrap.cc:492] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:12:57.526113 29029 ts_tablet_manager.cc:1367] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 14:12:57.527103 29029 raft_consensus.cc:348] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.525234 29031 ts_tablet_manager.cc:1367] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.012s
I20240503 14:12:57.527381 29029 raft_consensus.cc:374] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.527477 29029 raft_consensus.cc:729] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.527832 29029 consensus_queue.cc:260] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.528502 29029 ts_tablet_manager.cc:1398] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:57.529844 29027 tablet_bootstrap.cc:492] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:12:57.530032 29027 ts_tablet_manager.cc:1367] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 14:12:57.530965 29027 raft_consensus.cc:348] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.531230 29027 raft_consensus.cc:374] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.531355 29027 raft_consensus.cc:729] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.531061 29028 raft_consensus.cc:348] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.531483 29028 raft_consensus.cc:374] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.531622 29028 raft_consensus.cc:729] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.531694 29027 consensus_queue.cc:260] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.532253 29027 ts_tablet_manager.cc:1398] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:57.532624 29028 consensus_queue.cc:260] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:12:57.533728 28350 tablet.cc:2369] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:57.534435 29028 ts_tablet_manager.cc:1398] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20240503 14:12:57.534893 29028 tablet_bootstrap.cc:492] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:12:57.537101 29028 tablet_bootstrap.cc:654] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.537068 29031 raft_consensus.cc:348] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.537497 29031 raft_consensus.cc:374] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.537618 29031 raft_consensus.cc:729] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.538347 29031 consensus_queue.cc:260] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.540253 29028 tablet_bootstrap.cc:492] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:12:57.540391 29028 ts_tablet_manager.cc:1367] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240503 14:12:57.540827 29031 ts_tablet_manager.cc:1398] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20240503 14:12:57.541218 29031 tablet_bootstrap.cc:492] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:12:57.541203 29028 raft_consensus.cc:348] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.541396 29028 raft_consensus.cc:374] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.541491 29028 raft_consensus.cc:729] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.541755 29028 consensus_queue.cc:260] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.542196 29028 ts_tablet_manager.cc:1398] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:57.542860 29031 tablet_bootstrap.cc:654] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.544818 29031 tablet_bootstrap.cc:492] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:12:57.544955 29031 ts_tablet_manager.cc:1367] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:57.545667 29031 raft_consensus.cc:348] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.545806 29031 raft_consensus.cc:374] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.545892 29031 raft_consensus.cc:729] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.546111 29031 consensus_queue.cc:260] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.546581 29031 ts_tablet_manager.cc:1398] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:57.546835 29031 tablet_bootstrap.cc:492] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:12:57.548264 29031 tablet_bootstrap.cc:654] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.550024 29031 tablet_bootstrap.cc:492] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:12:57.550148 29031 ts_tablet_manager.cc:1367] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:57.550653 29031 raft_consensus.cc:348] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.550808 29031 raft_consensus.cc:374] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.550863 29031 raft_consensus.cc:729] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.551016 29031 consensus_queue.cc:260] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.551436 29031 ts_tablet_manager.cc:1398] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:12:57.583525 29035 raft_consensus.cc:482] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:57.583789 29035 raft_consensus.cc:504] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.585443 29035 leader_election.cc:288] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:12:57.593025 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5867eb31f7344b178db1a2d0a22da70b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:12:57.593139 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5867eb31f7344b178db1a2d0a22da70b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:12:57.593592 28174 raft_consensus.cc:2454] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:12:57.593734 28043 raft_consensus.cc:2454] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:12:57.594468 28238 leader_election.cc:302] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:12:57.594902 29035 raft_consensus.cc:2781] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:57.595063 29035 raft_consensus.cc:482] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:57.595167 29035 raft_consensus.cc:3037] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.597232 29035 raft_consensus.cc:504] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.597790 29035 leader_election.cc:288] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:12:57.598309 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5867eb31f7344b178db1a2d0a22da70b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:12:57.598346 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5867eb31f7344b178db1a2d0a22da70b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:12:57.598493 28174 raft_consensus.cc:3037] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.598555 28043 raft_consensus.cc:3037] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.600665 28174 raft_consensus.cc:2454] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:12:57.601008 28238 leader_election.cc:302] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:12:57.601258 29035 raft_consensus.cc:2781] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:57.601521 28043 raft_consensus.cc:2454] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:12:57.601902 29035 raft_consensus.cc:686] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:12:57.602383 29035 consensus_queue.cc:237] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
W20240503 14:12:57.603914 28220 tablet.cc:2369] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:57.606762 27900 catalog_manager.cc:5484] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:12:57.630810 29041 raft_consensus.cc:482] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:57.631042 29041 raft_consensus.cc:504] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.632669 29041 leader_election.cc:288] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:12:57.640658 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5031adcf1b4e55a73b615080fbd73c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:12:57.641229 28434 raft_consensus.cc:2454] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:12:57.641086 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5031adcf1b4e55a73b615080fbd73c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:12:57.641479 28304 raft_consensus.cc:2454] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:12:57.641855 27979 leader_election.cc:302] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:12:57.642180 29041 raft_consensus.cc:2781] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:57.642326 29041 raft_consensus.cc:482] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:57.642439 29041 raft_consensus.cc:3037] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.644344 29041 raft_consensus.cc:504] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.644889 29041 leader_election.cc:288] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:12:57.645303 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5031adcf1b4e55a73b615080fbd73c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:12:57.645462 28434 raft_consensus.cc:3037] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.645445 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa5031adcf1b4e55a73b615080fbd73c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:12:57.645637 28304 raft_consensus.cc:3037] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.647522 28434 raft_consensus.cc:2454] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:12:57.647660 28304 raft_consensus.cc:2454] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:12:57.647868 27979 leader_election.cc:302] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:12:57.648169 29041 raft_consensus.cc:2781] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:57.648569 29041 raft_consensus.cc:686] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:12:57.649011 29041 consensus_queue.cc:237] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:12:57.652938 27900 catalog_manager.cc:5484] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
W20240503 14:12:57.655822 28089 tablet.cc:2369] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:12:57.726590 28480 tablet.cc:2369] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:57.789245 29041 raft_consensus.cc:482] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:57.789535 29041 raft_consensus.cc:504] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.790388 29041 leader_election.cc:288] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:12:57.790946 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26fbb09f31c4c8f81788c784da6f205" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:12:57.791219 28434 raft_consensus.cc:2454] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:12:57.791678 27979 leader_election.cc:302] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:12:57.791977 29041 raft_consensus.cc:2781] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:57.792057 29041 raft_consensus.cc:482] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:57.792161 29041 raft_consensus.cc:3037] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.794078 29041 raft_consensus.cc:504] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.794715 29041 leader_election.cc:288] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:12:57.795233 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26fbb09f31c4c8f81788c784da6f205" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:12:57.795523 28434 raft_consensus.cc:3037] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.796710 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26fbb09f31c4c8f81788c784da6f205" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:12:57.796788 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c26fbb09f31c4c8f81788c784da6f205" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:12:57.796978 28174 raft_consensus.cc:2454] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:12:57.797878 28434 raft_consensus.cc:2454] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:12:57.798200 27979 leader_election.cc:302] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6409961ee8c0445687c11724213c44a9
I20240503 14:12:57.798463 29041 raft_consensus.cc:2781] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:57.798625 29041 raft_consensus.cc:686] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:12:57.798882 29041 consensus_queue.cc:237] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:57.801568 27900 catalog_manager.cc:5484] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:12:58.066393 29035 consensus_queue.cc:1035] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:58.072696 29046 consensus_queue.cc:1035] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:58.096800 29041 consensus_queue.cc:1035] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:58.102222 29050 consensus_queue.cc:1035] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:58.138468 29053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:58.138773 29053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:58.146093 29053 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:58.177258 28173 raft_consensus.cc:3037] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.179870 29050 consensus_queue.cc:1035] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:58.184320 29050 consensus_queue.cc:1035] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 257.896 ms
time per row: 0.128948 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:59.485234 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:59.493983 28284 tablet_service.cc:1467] Processing CreateTablet for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.494489 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.494416 28414 tablet_service.cc:1467] Processing CreateTablet for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.494650 28154 tablet_service.cc:1467] Processing CreateTablet for tablet d7b192797b6c40d4853e6ae4583ef3ca (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.494850 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.495043 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b192797b6c40d4853e6ae4583ef3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.496054 28152 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddd3f269b35481baa313da70ee987de (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.496433 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddd3f269b35481baa313da70ee987de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.500775 28153 tablet_service.cc:1467] Processing CreateTablet for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.501180 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.504894 28283 tablet_service.cc:1467] Processing CreateTablet for tablet d7b192797b6c40d4853e6ae4583ef3ca (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.505182 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddd3f269b35481baa313da70ee987de (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.505286 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b192797b6c40d4853e6ae4583ef3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.505565 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddd3f269b35481baa313da70ee987de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.505622 28413 tablet_service.cc:1467] Processing CreateTablet for tablet d7b192797b6c40d4853e6ae4583ef3ca (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.506016 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b192797b6c40d4853e6ae4583ef3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.510867 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddd3f269b35481baa313da70ee987de (DEFAULT_TABLE table=rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.511267 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ddd3f269b35481baa313da70ee987de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.515892 29116 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:12:59.516448 29119 tablet_bootstrap.cc:492] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:12:59.517791 29119 tablet_bootstrap.cc:654] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.518850 29118 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:12:59.519192 29117 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:12:59.519748 29116 tablet_bootstrap.cc:654] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.520146 29118 tablet_bootstrap.cc:654] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.520879 29117 tablet_bootstrap.cc:654] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.521637 29116 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:12:59.522306 29116 ts_tablet_manager.cc:1367] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:12:59.523116 29119 tablet_bootstrap.cc:492] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:12:59.523247 29119 ts_tablet_manager.cc:1367] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:12:59.523211 29116 raft_consensus.cc:348] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.523360 29118 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:12:59.523427 29116 raft_consensus.cc:374] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.523488 29118 ts_tablet_manager.cc:1367] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:59.523522 29116 raft_consensus.cc:729] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.523775 29116 consensus_queue.cc:260] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.524246 29118 raft_consensus.cc:348] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.523931 29119 raft_consensus.cc:348] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.524447 29119 raft_consensus.cc:374] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.524470 29116 ts_tablet_manager.cc:1398] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:59.524629 29119 raft_consensus.cc:729] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.524879 29119 consensus_queue.cc:260] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.525104 29117 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:12:59.525296 29117 ts_tablet_manager.cc:1367] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 14:12:59.525449 29119 ts_tablet_manager.cc:1398] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:59.526099 29117 raft_consensus.cc:348] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.526355 29117 raft_consensus.cc:374] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.526455 29117 raft_consensus.cc:729] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.526688 29117 consensus_queue.cc:260] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.526912 29116 tablet_bootstrap.cc:492] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:12:59.527124 29118 raft_consensus.cc:374] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.527233 29118 raft_consensus.cc:729] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.527487 29118 consensus_queue.cc:260] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.528260 29117 ts_tablet_manager.cc:1398] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:59.528602 29117 tablet_bootstrap.cc:492] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:12:59.528847 29116 tablet_bootstrap.cc:654] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.530270 29117 tablet_bootstrap.cc:654] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.535373 29117 tablet_bootstrap.cc:492] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:12:59.535545 29117 ts_tablet_manager.cc:1367] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 14:12:59.536202 29117 raft_consensus.cc:348] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.536474 29117 raft_consensus.cc:374] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.536500 29116 tablet_bootstrap.cc:492] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:12:59.536636 29117 raft_consensus.cc:729] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.536655 29116 ts_tablet_manager.cc:1367] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240503 14:12:59.536971 29117 consensus_queue.cc:260] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.537508 29117 ts_tablet_manager.cc:1398] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:59.537411 29116 raft_consensus.cc:348] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.537618 29116 raft_consensus.cc:374] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.537631 29118 ts_tablet_manager.cc:1398] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.014s user 0.002s sys 0.003s
I20240503 14:12:59.537753 29116 raft_consensus.cc:729] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.538036 29117 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:12:59.538492 29118 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:12:59.539498 29117 tablet_bootstrap.cc:654] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.538029 29116 consensus_queue.cc:260] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.540722 29118 tablet_bootstrap.cc:654] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.541044 29116 ts_tablet_manager.cc:1398] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:59.541416 29116 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:12:59.542058 29117 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:12:59.542274 29117 ts_tablet_manager.cc:1367] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:59.542968 29116 tablet_bootstrap.cc:654] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.543197 29117 raft_consensus.cc:348] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.544123 29117 raft_consensus.cc:374] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.544279 29117 raft_consensus.cc:729] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.544582 29117 consensus_queue.cc:260] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.545238 29117 ts_tablet_manager.cc:1398] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:59.547761 29118 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:12:59.547919 29118 ts_tablet_manager.cc:1367] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240503 14:12:59.548454 29118 raft_consensus.cc:348] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.548630 29118 raft_consensus.cc:374] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.548712 29118 raft_consensus.cc:729] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.548945 29118 consensus_queue.cc:260] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.549315 29118 ts_tablet_manager.cc:1398] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:59.550475 29116 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:12:59.550606 29116 ts_tablet_manager.cc:1367] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.005s
I20240503 14:12:59.551213 29116 raft_consensus.cc:348] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.551822 29116 raft_consensus.cc:374] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.551931 29116 raft_consensus.cc:729] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.552141 29116 consensus_queue.cc:260] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.553141 29116 ts_tablet_manager.cc:1398] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:59.562307 29124 raft_consensus.cc:482] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:59.562481 29124 raft_consensus.cc:504] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.564163 29124 leader_election.cc:288] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:12:59.571592 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd3f269b35481baa313da70ee987de" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:12:59.571808 28304 raft_consensus.cc:2454] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:12:59.571926 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd3f269b35481baa313da70ee987de" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:12:59.572144 28042 raft_consensus.cc:2454] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:12:59.572300 28110 leader_election.cc:302] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:12:59.572633 29124 raft_consensus.cc:2781] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:59.572743 29124 raft_consensus.cc:482] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:59.572845 29124 raft_consensus.cc:3037] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.574936 29124 raft_consensus.cc:504] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.575438 29124 leader_election.cc:288] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:12:59.575894 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd3f269b35481baa313da70ee987de" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:12:59.575999 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddd3f269b35481baa313da70ee987de" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:12:59.576108 28304 raft_consensus.cc:3037] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.576179 28042 raft_consensus.cc:3037] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.578158 28304 raft_consensus.cc:2454] T 8ddd3f269b35481baa313da70ee987de P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:12:59.578172 28042 raft_consensus.cc:2454] T 8ddd3f269b35481baa313da70ee987de P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:12:59.578526 28110 leader_election.cc:302] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:12:59.578775 29124 raft_consensus.cc:2781] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:59.579190 29124 raft_consensus.cc:686] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:12:59.579663 29124 consensus_queue.cc:237] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.582456 27893 catalog_manager.cc:5484] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:12:59.696669 29125 raft_consensus.cc:482] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:59.696857 29125 raft_consensus.cc:504] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.698395 29125 leader_election.cc:288] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:12:59.706768 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08c6dba4dc4e90ab11a155ec6ecdfd" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:12:59.707048 28304 raft_consensus.cc:2454] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:12:59.707005 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08c6dba4dc4e90ab11a155ec6ecdfd" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:12:59.707214 28173 raft_consensus.cc:2454] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:12:59.707530 28370 leader_election.cc:302] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:12:59.707902 29125 raft_consensus.cc:2781] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:59.708017 29125 raft_consensus.cc:482] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:59.708115 29125 raft_consensus.cc:3037] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.710008 29125 raft_consensus.cc:504] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.710549 29125 leader_election.cc:288] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:12:59.710990 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08c6dba4dc4e90ab11a155ec6ecdfd" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:12:59.711035 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac08c6dba4dc4e90ab11a155ec6ecdfd" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:12:59.711208 28304 raft_consensus.cc:3037] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.711208 28173 raft_consensus.cc:3037] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.713966 28304 raft_consensus.cc:2454] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:12:59.713966 28173 raft_consensus.cc:2454] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:12:59.714414 28370 leader_election.cc:302] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:12:59.714846 29125 raft_consensus.cc:2781] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:59.715314 29125 raft_consensus.cc:686] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:12:59.715751 29125 consensus_queue.cc:237] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:12:59.716665 29046 raft_consensus.cc:482] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:59.716810 29046 raft_consensus.cc:504] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.717630 29046 leader_election.cc:288] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:12:59.718135 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b192797b6c40d4853e6ae4583ef3ca" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:12:59.718464 28173 raft_consensus.cc:2454] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:12:59.719185 28238 leader_election.cc:302] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:12:59.719101 27893 catalog_manager.cc:5484] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:12:59.719528 29046 raft_consensus.cc:2781] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:59.719621 29046 raft_consensus.cc:482] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:59.719745 29046 raft_consensus.cc:3037] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.722465 29046 raft_consensus.cc:504] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.723066 29046 leader_election.cc:288] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:12:59.723443 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b192797b6c40d4853e6ae4583ef3ca" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:12:59.723614 28173 raft_consensus.cc:3037] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.726040 28173 raft_consensus.cc:2454] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:12:59.726119 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b192797b6c40d4853e6ae4583ef3ca" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:12:59.726209 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b192797b6c40d4853e6ae4583ef3ca" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:12:59.726373 28434 raft_consensus.cc:2454] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:12:59.726485 28238 leader_election.cc:302] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:12:59.726776 29046 raft_consensus.cc:2781] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:59.727222 29046 raft_consensus.cc:686] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:12:59.727516 29046 consensus_queue.cc:237] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:12:59.729754 27893 catalog_manager.cc:5484] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:12:59.993707 29124 consensus_queue.cc:1035] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.001017 29150 consensus_queue.cc:1035] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:00.078415 29146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:00.078758 29146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:00.086292 29146 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240503 14:13:00.205678 28173 raft_consensus.cc:1261] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:00.208317 28433 raft_consensus.cc:3037] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.211642 28433 raft_consensus.cc:1261] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:00.213560 29046 consensus_queue.cc:1035] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.215991 29144 consensus_queue.cc:1035] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.216387 29125 consensus_queue.cc:1035] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.221784 29125 consensus_queue.cc:1035] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.251456 29182 mvcc.cc:205] Tried to move back new op lower bound from 7023597896503558144 to 7023597894565953536. Current Snapshot: MvccSnapshot[applied={T|T < 7023597896503558144}]
I20240503 14:13:00.256750 29186 mvcc.cc:205] Tried to move back new op lower bound from 7023597896503558144 to 7023597894565953536. Current Snapshot: MvccSnapshot[applied={T|T < 7023597896503558144}]
I20240503 14:13:00.301470 29148 mvcc.cc:205] Tried to move back new op lower bound from 7023597896503558144 to 7023597894565953536. Current Snapshot: MvccSnapshot[applied={T|T < 7023597896503558144}]
INSERT report
rows total: 2000
time total: 220.194 ms
time per row: 0.110097 ms
I20240503 14:13:00.447108 27893 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:00.455675 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 4d05336747e94520bccc53cdb85e51b7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.456164 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d05336747e94520bccc53cdb85e51b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.456056 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 4d05336747e94520bccc53cdb85e51b7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.456275 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 4d05336747e94520bccc53cdb85e51b7 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.456534 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d05336747e94520bccc53cdb85e51b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.456676 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d05336747e94520bccc53cdb85e51b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.459802 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 07f07ce659e844e3afc090a0753c1128 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.460175 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f07ce659e844e3afc090a0753c1128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.463445 28283 tablet_service.cc:1467] Processing CreateTablet for tablet ef0b6bb524e842f5ade540e691e50a97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.463835 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0b6bb524e842f5ade540e691e50a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.467096 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 07f07ce659e844e3afc090a0753c1128 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.467547 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f07ce659e844e3afc090a0753c1128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.467815 28022 tablet_service.cc:1467] Processing CreateTablet for tablet ef0b6bb524e842f5ade540e691e50a97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.468036 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 07f07ce659e844e3afc090a0753c1128 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.468205 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0b6bb524e842f5ade540e691e50a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.468370 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f07ce659e844e3afc090a0753c1128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.467082 28413 tablet_service.cc:1467] Processing CreateTablet for tablet ef0b6bb524e842f5ade540e691e50a97 (DEFAULT_TABLE table=rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.474509 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef0b6bb524e842f5ade540e691e50a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.475756 29208 tablet_bootstrap.cc:492] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:00.477449 29208 tablet_bootstrap.cc:654] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.485419 29209 tablet_bootstrap.cc:492] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:00.487655 29207 tablet_bootstrap.cc:492] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:00.488970 29209 tablet_bootstrap.cc:654] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.489163 29206 tablet_bootstrap.cc:492] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:00.490602 29207 tablet_bootstrap.cc:654] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.491339 29206 tablet_bootstrap.cc:654] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.495477 29208 tablet_bootstrap.cc:492] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:00.495621 29208 ts_tablet_manager.cc:1367] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.008s
I20240503 14:13:00.495954 29206 tablet_bootstrap.cc:492] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:00.496229 29206 ts_tablet_manager.cc:1367] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:13:00.496382 29208 raft_consensus.cc:348] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.496595 29208 raft_consensus.cc:374] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.496695 29208 raft_consensus.cc:729] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.497001 29208 consensus_queue.cc:260] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.497535 29208 ts_tablet_manager.cc:1398] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:00.497690 29206 raft_consensus.cc:348] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.497886 29208 tablet_bootstrap.cc:492] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:00.498039 29206 raft_consensus.cc:374] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.498195 29206 raft_consensus.cc:729] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.498610 29206 consensus_queue.cc:260] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.500044 29208 tablet_bootstrap.cc:654] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.500376 29206 ts_tablet_manager.cc:1398] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:00.500823 29206 tablet_bootstrap.cc:492] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:00.501821 29207 tablet_bootstrap.cc:492] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:00.502023 29207 ts_tablet_manager.cc:1367] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240503 14:13:00.502861 29207 raft_consensus.cc:348] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.503118 29207 raft_consensus.cc:374] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.503235 29207 raft_consensus.cc:729] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.503283 29206 tablet_bootstrap.cc:654] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.503566 29207 consensus_queue.cc:260] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.504684 29207 ts_tablet_manager.cc:1398] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:00.504976 29207 tablet_bootstrap.cc:492] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:00.505858 29208 tablet_bootstrap.cc:492] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:00.506064 29208 ts_tablet_manager.cc:1367] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:13:00.506923 29208 raft_consensus.cc:348] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.507107 29207 tablet_bootstrap.cc:654] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.507347 29208 raft_consensus.cc:374] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.507480 29208 raft_consensus.cc:729] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.507797 29208 consensus_queue.cc:260] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.509285 29209 tablet_bootstrap.cc:492] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:00.509343 29208 ts_tablet_manager.cc:1398] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:00.509464 29209 ts_tablet_manager.cc:1367] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.008s
I20240503 14:13:00.510294 29209 raft_consensus.cc:348] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.510550 29209 raft_consensus.cc:374] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.510679 29209 raft_consensus.cc:729] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.510968 29209 consensus_queue.cc:260] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.512349 29209 ts_tablet_manager.cc:1398] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:00.513243 29207 tablet_bootstrap.cc:492] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:00.513427 29207 ts_tablet_manager.cc:1367] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240503 14:13:00.514186 29207 raft_consensus.cc:348] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.514463 29207 raft_consensus.cc:374] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.514564 29207 raft_consensus.cc:729] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.514803 29207 consensus_queue.cc:260] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.515470 29207 ts_tablet_manager.cc:1398] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:00.515763 29207 tablet_bootstrap.cc:492] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:00.517167 29207 tablet_bootstrap.cc:654] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.518957 29206 tablet_bootstrap.cc:492] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:00.519174 29206 ts_tablet_manager.cc:1367] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.005s
I20240503 14:13:00.520044 29206 raft_consensus.cc:348] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.520274 29206 raft_consensus.cc:374] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.520390 29206 raft_consensus.cc:729] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.520646 29206 consensus_queue.cc:260] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.520999 29207 tablet_bootstrap.cc:492] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:00.521070 29206 ts_tablet_manager.cc:1398] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:00.521137 29207 ts_tablet_manager.cc:1367] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:00.521375 29206 tablet_bootstrap.cc:492] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:00.521860 29207 raft_consensus.cc:348] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.522048 29207 raft_consensus.cc:374] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.522156 29207 raft_consensus.cc:729] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.522430 29207 consensus_queue.cc:260] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.522873 29206 tablet_bootstrap.cc:654] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.522862 29041 raft_consensus.cc:482] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:00.523110 29135 raft_consensus.cc:482] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:00.523017 29041 raft_consensus.cc:504] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.523247 29135 raft_consensus.cc:504] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.523487 29207 ts_tablet_manager.cc:1398] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:00.523686 29041 leader_election.cc:288] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:00.524081 29135 leader_election.cc:288] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:00.524389 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d05336747e94520bccc53cdb85e51b7" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:00.524653 28434 raft_consensus.cc:2454] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:00.524812 29206 tablet_bootstrap.cc:492] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:00.524996 29206 ts_tablet_manager.cc:1367] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:00.525456 27979 leader_election.cc:302] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:00.525931 29206 raft_consensus.cc:348] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.526157 29206 raft_consensus.cc:374] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.526227 29041 raft_consensus.cc:2781] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:00.526314 29206 raft_consensus.cc:729] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.526371 29041 raft_consensus.cc:482] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:00.526499 29041 raft_consensus.cc:3037] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.526522 29206 consensus_queue.cc:260] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.527179 29206 ts_tablet_manager.cc:1398] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:00.529788 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f07ce659e844e3afc090a0753c1128" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:00.529875 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d05336747e94520bccc53cdb85e51b7" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:00.530007 28042 raft_consensus.cc:2454] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:00.530077 28174 raft_consensus.cc:2454] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:00.531030 29041 raft_consensus.cc:504] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.531170 28110 leader_election.cc:302] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:00.531452 29135 raft_consensus.cc:2781] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:00.531585 29135 raft_consensus.cc:482] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:00.531693 29135 raft_consensus.cc:3037] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.532188 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d05336747e94520bccc53cdb85e51b7" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:00.532362 28434 raft_consensus.cc:3037] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.532716 29041 leader_election.cc:288] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:00.534615 29135 raft_consensus.cc:504] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.535276 29135 leader_election.cc:288] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:00.535259 28434 raft_consensus.cc:2454] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:00.535826 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d05336747e94520bccc53cdb85e51b7" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:00.535918 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f07ce659e844e3afc090a0753c1128" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:00.536031 28174 raft_consensus.cc:3037] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.536105 28042 raft_consensus.cc:3037] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.537854 27979 leader_election.cc:302] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:00.538182 29041 raft_consensus.cc:2781] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:00.538399 29041 raft_consensus.cc:686] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:00.538853 28174 raft_consensus.cc:2454] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:00.538857 29041 consensus_queue.cc:237] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.538972 28042 raft_consensus.cc:2454] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:00.539531 28110 leader_election.cc:302] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:00.539608 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f07ce659e844e3afc090a0753c1128" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:00.539836 28434 raft_consensus.cc:3037] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.539903 29135 raft_consensus.cc:2781] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:00.540187 29135 raft_consensus.cc:686] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:00.540495 29135 consensus_queue.cc:237] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:00.542802 28434 raft_consensus.cc:2454] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:00.544569 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f07ce659e844e3afc090a0753c1128" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:00.545293 28433 raft_consensus.cc:2362] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1. Re-sending same reply.
I20240503 14:13:00.546519 27893 catalog_manager.cc:5484] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:00.546828 27899 catalog_manager.cc:5484] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:00.730887 29196 raft_consensus.cc:482] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:00.731058 29196 raft_consensus.cc:504] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.731782 29196 leader_election.cc:288] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:00.732962 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0b6bb524e842f5ade540e691e50a97" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:00.733191 28304 raft_consensus.cc:2454] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:00.733743 28370 leader_election.cc:302] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:00.734069 29196 raft_consensus.cc:2781] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:00.734194 29196 raft_consensus.cc:482] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:00.734323 29196 raft_consensus.cc:3037] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.736865 29196 raft_consensus.cc:504] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.737439 29196 leader_election.cc:288] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:00.737915 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0b6bb524e842f5ade540e691e50a97" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:00.738145 28304 raft_consensus.cc:3037] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.739105 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0b6bb524e842f5ade540e691e50a97" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:00.739229 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef0b6bb524e842f5ade540e691e50a97" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:00.739311 28042 raft_consensus.cc:2454] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:00.740311 28304 raft_consensus.cc:2454] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:00.740619 28370 leader_election.cc:302] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:13:00.740952 29196 raft_consensus.cc:2781] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:00.741111 29196 raft_consensus.cc:686] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:00.741374 29196 consensus_queue.cc:237] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:00.744055 27899 catalog_manager.cc:5484] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:00.939544 29041 consensus_queue.cc:1035] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.948155 29041 consensus_queue.cc:1035] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:01.091140 29227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:01.091465 29227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:01.098562 29227 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:01.119663 29135 consensus_queue.cc:1035] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:01.124869 29135 consensus_queue.cc:1035] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240503 14:13:01.181501 29196 consensus_queue.cc:1035] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:01.188395 28043 raft_consensus.cc:3037] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.192883 29196 consensus_queue.cc:1035] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 238.36 ms
time per row: 0.11918 ms
I20240503 14:13:01.479247 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:01.488070 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 88f88e7eb7434450af76714256729887 (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.488324 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 88f88e7eb7434450af76714256729887 (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.488555 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f88e7eb7434450af76714256729887. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.488481 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 88f88e7eb7434450af76714256729887 (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.488711 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f88e7eb7434450af76714256729887. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.488935 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f88e7eb7434450af76714256729887. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.491837 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 400ce95594a743108bb4ee1e64458956 (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.491966 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 519fed0fc03b4b9788803aeb0bbcfdad (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.492264 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400ce95594a743108bb4ee1e64458956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.492519 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519fed0fc03b4b9788803aeb0bbcfdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.493311 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 400ce95594a743108bb4ee1e64458956 (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.493300 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 400ce95594a743108bb4ee1e64458956 (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.493687 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400ce95594a743108bb4ee1e64458956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.493741 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400ce95594a743108bb4ee1e64458956. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.498307 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 519fed0fc03b4b9788803aeb0bbcfdad (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.498728 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519fed0fc03b4b9788803aeb0bbcfdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.498629 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 519fed0fc03b4b9788803aeb0bbcfdad (DEFAULT_TABLE table=rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.499123 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519fed0fc03b4b9788803aeb0bbcfdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.518911 29297 tablet_bootstrap.cc:492] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:01.521092 29297 tablet_bootstrap.cc:654] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.521759 29299 tablet_bootstrap.cc:492] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:01.522213 29298 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:01.522581 29296 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:01.523692 29299 tablet_bootstrap.cc:654] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.524673 29298 tablet_bootstrap.cc:654] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.533114 29299 tablet_bootstrap.cc:492] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:01.533252 29299 ts_tablet_manager.cc:1367] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 14:13:01.533493 29296 tablet_bootstrap.cc:654] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.534124 29299 raft_consensus.cc:348] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.534526 29299 raft_consensus.cc:374] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.534716 29299 raft_consensus.cc:729] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.535108 29299 consensus_queue.cc:260] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.536127 29299 ts_tablet_manager.cc:1398] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:01.536501 29299 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:01.538141 29299 tablet_bootstrap.cc:654] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.539487 29296 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:01.539722 29296 ts_tablet_manager.cc:1367] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 14:13:01.540722 29296 raft_consensus.cc:348] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.541152 29296 raft_consensus.cc:374] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.541316 29296 raft_consensus.cc:729] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.541680 29296 consensus_queue.cc:260] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.542488 29296 ts_tablet_manager.cc:1398] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:01.542850 29296 tablet_bootstrap.cc:492] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:01.543359 29299 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:01.543551 29299 ts_tablet_manager.cc:1367] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 14:13:01.544302 29299 raft_consensus.cc:348] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.544483 29296 tablet_bootstrap.cc:654] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.544561 29299 raft_consensus.cc:374] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.544694 29299 raft_consensus.cc:729] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.545003 29299 consensus_queue.cc:260] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.545653 29299 ts_tablet_manager.cc:1398] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:01.546484 29297 tablet_bootstrap.cc:492] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:01.546623 29297 ts_tablet_manager.cc:1367] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.020s
I20240503 14:13:01.547505 29298 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:01.547396 29297 raft_consensus.cc:348] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.547605 29297 raft_consensus.cc:374] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.547708 29297 raft_consensus.cc:729] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.547871 29298 ts_tablet_manager.cc:1367] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.005s
I20240503 14:13:01.547946 29297 consensus_queue.cc:260] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.548961 29298 raft_consensus.cc:348] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.549335 29298 raft_consensus.cc:374] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.549458 29298 raft_consensus.cc:729] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.549695 29298 consensus_queue.cc:260] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.550361 29298 ts_tablet_manager.cc:1398] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:01.550724 29298 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:01.551017 29296 tablet_bootstrap.cc:492] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:01.551167 29296 ts_tablet_manager.cc:1367] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:13:01.551824 29296 raft_consensus.cc:348] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.552029 29296 raft_consensus.cc:374] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.552114 29296 raft_consensus.cc:729] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.552333 29296 consensus_queue.cc:260] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.552861 29296 ts_tablet_manager.cc:1398] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:01.553014 29298 tablet_bootstrap.cc:654] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.553185 29297 ts_tablet_manager.cc:1398] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:01.553457 29297 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:01.555140 29296 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:01.556188 29297 tablet_bootstrap.cc:654] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.557543 29296 tablet_bootstrap.cc:654] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.572921 29297 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:01.573107 29297 ts_tablet_manager.cc:1367] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.014s
I20240503 14:13:01.573257 29296 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:01.573413 29296 ts_tablet_manager.cc:1367] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.012s
I20240503 14:13:01.573480 29298 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:01.573722 29298 ts_tablet_manager.cc:1367] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.015s
I20240503 14:13:01.573870 29297 raft_consensus.cc:348] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.574050 29297 raft_consensus.cc:374] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.574132 29297 raft_consensus.cc:729] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.574203 29296 raft_consensus.cc:348] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.574431 29296 raft_consensus.cc:374] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.574409 29297 consensus_queue.cc:260] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.574537 29296 raft_consensus.cc:729] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.574517 29298 raft_consensus.cc:348] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.574757 29298 raft_consensus.cc:374] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.574739 29296 consensus_queue.cc:260] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.574884 29298 raft_consensus.cc:729] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.575106 29297 ts_tablet_manager.cc:1398] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:01.575435 29296 ts_tablet_manager.cc:1398] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:01.576032 29298 consensus_queue.cc:260] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.576617 29298 ts_tablet_manager.cc:1398] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:01.618790 29144 raft_consensus.cc:482] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:01.619002 29144 raft_consensus.cc:504] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.619809 29144 leader_election.cc:288] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:01.620318 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519fed0fc03b4b9788803aeb0bbcfdad" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:01.620380 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519fed0fc03b4b9788803aeb0bbcfdad" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:01.620546 28174 raft_consensus.cc:2454] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:01.620602 28433 raft_consensus.cc:2454] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:01.621012 28238 leader_election.cc:302] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:01.621296 29144 raft_consensus.cc:2781] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:01.621392 29144 raft_consensus.cc:482] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:01.621508 29144 raft_consensus.cc:3037] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.623493 29144 raft_consensus.cc:504] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.624008 29144 leader_election.cc:288] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:01.624428 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519fed0fc03b4b9788803aeb0bbcfdad" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:01.624503 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "519fed0fc03b4b9788803aeb0bbcfdad" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:01.624620 28174 raft_consensus.cc:3037] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.624718 28433 raft_consensus.cc:3037] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.627491 28174 raft_consensus.cc:2454] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:01.627532 28433 raft_consensus.cc:2454] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:01.627915 28240 leader_election.cc:302] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:01.628521 29144 raft_consensus.cc:2781] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:01.628994 29144 raft_consensus.cc:686] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:01.629345 29144 consensus_queue.cc:237] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.635210 27901 catalog_manager.cc:5484] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:01.680423 29144 raft_consensus.cc:482] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:01.680634 29144 raft_consensus.cc:504] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.681313 29144 leader_election.cc:288] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:01.681854 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f88e7eb7434450af76714256729887" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:01.681874 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f88e7eb7434450af76714256729887" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:01.682097 28174 raft_consensus.cc:2454] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:01.682164 28043 raft_consensus.cc:2454] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:01.682579 28238 leader_election.cc:302] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:01.682868 29144 raft_consensus.cc:2781] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:01.682986 29144 raft_consensus.cc:482] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:01.683068 29144 raft_consensus.cc:3037] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.684965 29144 raft_consensus.cc:504] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.685596 29144 leader_election.cc:288] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:01.686039 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f88e7eb7434450af76714256729887" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:01.686039 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f88e7eb7434450af76714256729887" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:01.686295 28174 raft_consensus.cc:3037] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.686295 28043 raft_consensus.cc:3037] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.689100 28043 raft_consensus.cc:2454] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:01.689098 28174 raft_consensus.cc:2454] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:01.689502 28238 leader_election.cc:302] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:01.689832 29144 raft_consensus.cc:2781] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:01.689965 29144 raft_consensus.cc:686] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:01.690227 29144 consensus_queue.cc:237] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:01.692868 27901 catalog_manager.cc:5484] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:01.851310 29144 raft_consensus.cc:482] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:01.851531 29144 raft_consensus.cc:504] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.852233 29144 leader_election.cc:288] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:01.852816 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400ce95594a743108bb4ee1e64458956" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:01.852797 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400ce95594a743108bb4ee1e64458956" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:01.853037 28043 raft_consensus.cc:2454] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:01.853037 28433 raft_consensus.cc:2454] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:01.853456 28240 leader_election.cc:302] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:01.853765 29144 raft_consensus.cc:2781] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:01.853899 29144 raft_consensus.cc:482] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:01.853999 29144 raft_consensus.cc:3037] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.855855 29144 raft_consensus.cc:504] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.856400 29144 leader_election.cc:288] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:01.856787 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400ce95594a743108bb4ee1e64458956" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:01.856902 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400ce95594a743108bb4ee1e64458956" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:01.857002 28043 raft_consensus.cc:3037] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.857108 28433 raft_consensus.cc:3037] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.859910 28043 raft_consensus.cc:2454] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:01.859899 28433 raft_consensus.cc:2454] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:01.860368 28240 leader_election.cc:302] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:01.860659 29144 raft_consensus.cc:2781] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:01.860848 29144 raft_consensus.cc:686] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:01.861137 29144 consensus_queue.cc:237] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:01.863761 27899 catalog_manager.cc:5484] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:02.028975 29144 consensus_queue.cc:1035] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:02.036691 29144 consensus_queue.cc:1035] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:02.137614 29144 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:02.142299 29144 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:02.205195 29313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:02.205580 29313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:02.212900 29313 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:13:02.308828 29144 consensus_queue.cc:1035] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:02.314983 29309 consensus_queue.cc:1035] T 400ce95594a743108bb4ee1e64458956 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 209.69 ms
time per row: 0.104845 ms
I20240503 14:13:02.568042 27893 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:02.576459 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 9c670ee10e964df492f103cfbf1ccad8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.576565 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 9c670ee10e964df492f103cfbf1ccad8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.576792 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 9c670ee10e964df492f103cfbf1ccad8 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.576941 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c670ee10e964df492f103cfbf1ccad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.577054 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c670ee10e964df492f103cfbf1ccad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.577179 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c670ee10e964df492f103cfbf1ccad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.578656 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.579071 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.580132 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 07bd28e4bc574526a170efca0060ea07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.580509 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bd28e4bc574526a170efca0060ea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.580724 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.579071 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 07bd28e4bc574526a170efca0060ea07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.581092 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.581159 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bd28e4bc574526a170efca0060ea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.582370 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.582710 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.582619 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 07bd28e4bc574526a170efca0060ea07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.582981 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07bd28e4bc574526a170efca0060ea07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.596307 29377 tablet_bootstrap.cc:492] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:02.600579 29377 tablet_bootstrap.cc:654] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.607731 29380 tablet_bootstrap.cc:492] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:02.609853 29380 tablet_bootstrap.cc:654] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.611388 29379 tablet_bootstrap.cc:492] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:02.614260 29379 tablet_bootstrap.cc:654] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.614933 29378 tablet_bootstrap.cc:492] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:02.617359 29378 tablet_bootstrap.cc:654] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.621449 29379 tablet_bootstrap.cc:492] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:02.621865 29379 ts_tablet_manager.cc:1367] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240503 14:13:02.623010 29379 raft_consensus.cc:348] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.623448 29379 raft_consensus.cc:374] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.623613 29379 raft_consensus.cc:729] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.624038 29379 consensus_queue.cc:260] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.624939 29379 ts_tablet_manager.cc:1398] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:02.626842 29379 tablet_bootstrap.cc:492] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:02.627436 29377 tablet_bootstrap.cc:492] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:02.627615 29377 ts_tablet_manager.cc:1367] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.018s
I20240503 14:13:02.628482 29379 tablet_bootstrap.cc:654] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.628388 29377 raft_consensus.cc:348] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.628603 29377 raft_consensus.cc:374] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.628693 29377 raft_consensus.cc:729] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.628919 29377 consensus_queue.cc:260] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.629562 29377 ts_tablet_manager.cc:1398] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:02.630661 29377 tablet_bootstrap.cc:492] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:02.631464 29378 tablet_bootstrap.cc:492] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:02.631613 29378 ts_tablet_manager.cc:1367] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 14:13:02.632198 29377 tablet_bootstrap.cc:654] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.632405 29378 raft_consensus.cc:348] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.632630 29378 raft_consensus.cc:374] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.632716 29378 raft_consensus.cc:729] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.632966 29378 consensus_queue.cc:260] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.634582 29378 ts_tablet_manager.cc:1398] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:02.634922 29380 tablet_bootstrap.cc:492] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:02.634992 29378 tablet_bootstrap.cc:492] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:02.635162 29380 ts_tablet_manager.cc:1367] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.015s
I20240503 14:13:02.636029 29380 raft_consensus.cc:348] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.636291 29380 raft_consensus.cc:374] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.636412 29380 raft_consensus.cc:729] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.636705 29380 consensus_queue.cc:260] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.637728 29378 tablet_bootstrap.cc:654] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.638121 29380 ts_tablet_manager.cc:1398] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:02.638444 29380 tablet_bootstrap.cc:492] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:02.640619 29380 tablet_bootstrap.cc:654] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.650153 29380 tablet_bootstrap.cc:492] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:02.650408 29380 ts_tablet_manager.cc:1367] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240503 14:13:02.651209 29380 raft_consensus.cc:348] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.651474 29380 raft_consensus.cc:374] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.651511 29379 tablet_bootstrap.cc:492] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:02.651585 29380 raft_consensus.cc:729] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.651736 29379 ts_tablet_manager.cc:1367] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.006s
I20240503 14:13:02.651903 29380 consensus_queue.cc:260] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.652511 29379 raft_consensus.cc:348] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.652769 29379 raft_consensus.cc:374] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.652868 29379 raft_consensus.cc:729] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.653066 29380 ts_tablet_manager.cc:1398] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:02.653157 29379 consensus_queue.cc:260] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.653698 29379 ts_tablet_manager.cc:1398] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:02.657904 29378 tablet_bootstrap.cc:492] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:02.658079 29378 ts_tablet_manager.cc:1367] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.008s
I20240503 14:13:02.658154 29380 tablet_bootstrap.cc:492] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:02.658437 29377 tablet_bootstrap.cc:492] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:02.658651 29377 ts_tablet_manager.cc:1367] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.004s
I20240503 14:13:02.658970 29378 raft_consensus.cc:348] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.659193 29378 raft_consensus.cc:374] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.659276 29378 raft_consensus.cc:729] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.659554 29378 consensus_queue.cc:260] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.660219 29378 ts_tablet_manager.cc:1398] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:02.659798 29380 tablet_bootstrap.cc:654] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.660147 29377 raft_consensus.cc:348] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.661479 29377 raft_consensus.cc:374] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.661577 29377 raft_consensus.cc:729] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.661850 29377 consensus_queue.cc:260] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.662472 29377 ts_tablet_manager.cc:1398] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:02.662846 29380 tablet_bootstrap.cc:492] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:02.662969 29380 ts_tablet_manager.cc:1367] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:02.663596 29380 raft_consensus.cc:348] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.663765 29380 raft_consensus.cc:374] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.663834 29380 raft_consensus.cc:729] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.664057 29380 consensus_queue.cc:260] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.664542 29380 ts_tablet_manager.cc:1398] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:02.701772 29041 raft_consensus.cc:482] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:02.701948 29041 raft_consensus.cc:504] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.702622 29041 leader_election.cc:288] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:02.703091 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c670ee10e964df492f103cfbf1ccad8" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:02.703153 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c670ee10e964df492f103cfbf1ccad8" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:02.703297 28174 raft_consensus.cc:2454] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:02.703354 28304 raft_consensus.cc:2454] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:02.703708 27977 leader_election.cc:302] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:02.703967 29041 raft_consensus.cc:2781] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:02.704098 29041 raft_consensus.cc:482] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:02.704190 29041 raft_consensus.cc:3037] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.706040 29041 raft_consensus.cc:504] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.706575 29041 leader_election.cc:288] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:02.707008 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c670ee10e964df492f103cfbf1ccad8" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:02.707002 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c670ee10e964df492f103cfbf1ccad8" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:02.707198 28174 raft_consensus.cc:3037] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.707198 28304 raft_consensus.cc:3037] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.709923 28304 raft_consensus.cc:2454] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:02.709923 28174 raft_consensus.cc:2454] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:02.710318 27977 leader_election.cc:302] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:02.710592 29041 raft_consensus.cc:2781] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:02.710705 29041 raft_consensus.cc:686] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:02.710978 29041 consensus_queue.cc:237] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:02.713426 27893 catalog_manager.cc:5484] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:02.735355 29283 raft_consensus.cc:482] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:02.735532 29283 raft_consensus.cc:504] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.736187 29283 leader_election.cc:288] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:02.736635 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5cc9c71a534b7d8c1180b6e1d69af9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:02.736685 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5cc9c71a534b7d8c1180b6e1d69af9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:02.736887 28174 raft_consensus.cc:2454] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:02.736904 28043 raft_consensus.cc:2454] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:02.737313 28370 leader_election.cc:302] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:02.737587 29283 raft_consensus.cc:2781] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:02.737691 29283 raft_consensus.cc:482] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:02.737784 29283 raft_consensus.cc:3037] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.739727 29283 raft_consensus.cc:504] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.740285 29283 leader_election.cc:288] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:02.740638 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5cc9c71a534b7d8c1180b6e1d69af9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:02.740749 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5cc9c71a534b7d8c1180b6e1d69af9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:02.740846 28043 raft_consensus.cc:3037] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.740957 28174 raft_consensus.cc:3037] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.742866 28043 raft_consensus.cc:2454] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:02.742998 28174 raft_consensus.cc:2454] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:02.743237 28370 leader_election.cc:302] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:02.743512 29283 raft_consensus.cc:2781] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:02.743883 29283 raft_consensus.cc:686] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:02.744171 29283 consensus_queue.cc:237] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:02.746968 27893 catalog_manager.cc:5484] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:02.766950 29283 raft_consensus.cc:482] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:02.767185 29283 raft_consensus.cc:504] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.767793 29283 leader_election.cc:288] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:02.768414 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bd28e4bc574526a170efca0060ea07" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:02.768397 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bd28e4bc574526a170efca0060ea07" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:02.768658 28043 raft_consensus.cc:2454] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:02.768677 28304 raft_consensus.cc:2454] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:02.769074 28370 leader_election.cc:302] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:02.769340 29283 raft_consensus.cc:2781] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:02.769441 29283 raft_consensus.cc:482] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:02.769560 29283 raft_consensus.cc:3037] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.771780 29283 raft_consensus.cc:504] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.772238 29283 leader_election.cc:288] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:02.772684 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bd28e4bc574526a170efca0060ea07" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:02.772743 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07bd28e4bc574526a170efca0060ea07" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:02.772917 28043 raft_consensus.cc:3037] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.772939 28304 raft_consensus.cc:3037] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.775813 28043 raft_consensus.cc:2454] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:02.775816 28304 raft_consensus.cc:2454] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:02.776175 28370 leader_election.cc:302] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:02.776389 29283 raft_consensus.cc:2781] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:02.776546 29283 raft_consensus.cc:686] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:02.776822 29283 consensus_queue.cc:237] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:02.779515 27893 catalog_manager.cc:5484] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:03.137957 29283 consensus_queue.cc:1035] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:03.143280 29283 consensus_queue.cc:1035] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:03.163532 29395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:03.163997 29395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:03.176321 29395 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:03.178043 29283 consensus_queue.cc:1035] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:03.182862 29283 consensus_queue.cc:1035] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240503 14:13:03.254320 29041 consensus_queue.cc:1035] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:03.280226 29041 consensus_queue.cc:1035] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 245.473 ms
time per row: 0.122736 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:04.525835 27895 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:04.535140 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 609db8c3f3b245eaa31c82915b61188f (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.535626 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609db8c3f3b245eaa31c82915b61188f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.535696 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 609db8c3f3b245eaa31c82915b61188f (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.536130 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609db8c3f3b245eaa31c82915b61188f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.536955 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 49d89642d5364479877c7ba4700fdcee (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.536964 28021 tablet_service.cc:1467] Processing CreateTablet for tablet a816d34a63b945b08eec771d9e7c2d55 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.537330 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d89642d5364479877c7ba4700fdcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.537348 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a816d34a63b945b08eec771d9e7c2d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.539709 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 609db8c3f3b245eaa31c82915b61188f (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.540179 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609db8c3f3b245eaa31c82915b61188f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.540336 28152 tablet_service.cc:1467] Processing CreateTablet for tablet a816d34a63b945b08eec771d9e7c2d55 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.541031 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a816d34a63b945b08eec771d9e7c2d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.543347 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 49d89642d5364479877c7ba4700fdcee (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.543725 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d89642d5364479877c7ba4700fdcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.546809 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 49d89642d5364479877c7ba4700fdcee (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.547201 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d89642d5364479877c7ba4700fdcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.550611 28282 tablet_service.cc:1467] Processing CreateTablet for tablet a816d34a63b945b08eec771d9e7c2d55 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.550987 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a816d34a63b945b08eec771d9e7c2d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.555992 29463 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:04.558116 29463 tablet_bootstrap.cc:654] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.559787 29462 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:04.561623 29462 tablet_bootstrap.cc:654] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.562695 29463 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:04.563004 29463 ts_tablet_manager.cc:1367] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:04.564013 29463 raft_consensus.cc:348] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.565382 29463 raft_consensus.cc:374] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.565675 29463 raft_consensus.cc:729] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.566310 29462 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:04.566514 29462 ts_tablet_manager.cc:1367] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:13:04.566787 29464 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:04.566134 29463 consensus_queue.cc:260] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.567348 29462 raft_consensus.cc:348] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.567605 29462 raft_consensus.cc:374] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.567711 29462 raft_consensus.cc:729] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.567950 29462 consensus_queue.cc:260] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.569309 29464 tablet_bootstrap.cc:654] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.570573 29462 ts_tablet_manager.cc:1398] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:04.571648 29463 ts_tablet_manager.cc:1398] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:04.572503 29462 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:04.574441 29462 tablet_bootstrap.cc:654] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.576704 29467 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:04.576884 29463 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:04.578305 29467 tablet_bootstrap.cc:654] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.578361 29463 tablet_bootstrap.cc:654] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.579797 29464 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:04.580559 29464 ts_tablet_manager.cc:1367] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240503 14:13:04.581580 29464 raft_consensus.cc:348] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.581847 29464 raft_consensus.cc:374] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.581961 29464 raft_consensus.cc:729] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.582283 29464 consensus_queue.cc:260] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.583242 29464 ts_tablet_manager.cc:1398] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:04.583530 29464 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:04.584170 29463 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:04.584335 29463 ts_tablet_manager.cc:1367] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240503 14:13:04.584894 29467 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:04.585043 29464 tablet_bootstrap.cc:654] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.585076 29467 ts_tablet_manager.cc:1367] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:13:04.585183 29463 raft_consensus.cc:348] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.585402 29463 raft_consensus.cc:374] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.585484 29463 raft_consensus.cc:729] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.585711 29463 consensus_queue.cc:260] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.586371 29463 ts_tablet_manager.cc:1398] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:04.587309 29462 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:04.587603 29462 ts_tablet_manager.cc:1367] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.009s
I20240503 14:13:04.587023 29467 raft_consensus.cc:348] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.587817 29467 raft_consensus.cc:374] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.587972 29467 raft_consensus.cc:729] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.588229 29467 consensus_queue.cc:260] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.588567 29462 raft_consensus.cc:348] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.588822 29462 raft_consensus.cc:374] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.588899 29462 raft_consensus.cc:729] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.589354 29467 ts_tablet_manager.cc:1398] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:04.589195 29462 consensus_queue.cc:260] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.590806 29462 ts_tablet_manager.cc:1398] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:04.591120 29462 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:04.591149 29464 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:04.591390 29464 ts_tablet_manager.cc:1367] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240503 14:13:04.592206 29464 raft_consensus.cc:348] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.592480 29464 raft_consensus.cc:374] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.592633 29464 raft_consensus.cc:729] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.592921 29464 consensus_queue.cc:260] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.593488 29464 ts_tablet_manager.cc:1398] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:04.593649 29462 tablet_bootstrap.cc:654] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.593978 29464 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:04.595567 29464 tablet_bootstrap.cc:654] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.596498 29462 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:04.596648 29462 ts_tablet_manager.cc:1367] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 14:13:04.597788 29462 raft_consensus.cc:348] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.598062 29462 raft_consensus.cc:374] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.598173 29462 raft_consensus.cc:729] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.598495 29462 consensus_queue.cc:260] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.599121 29462 ts_tablet_manager.cc:1398] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:04.600517 29464 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:04.600654 29464 ts_tablet_manager.cc:1367] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 14:13:04.601279 29464 raft_consensus.cc:348] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.601459 29464 raft_consensus.cc:374] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.601537 29464 raft_consensus.cc:729] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.601720 29464 consensus_queue.cc:260] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.602209 29464 ts_tablet_manager.cc:1398] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:04.635933 29457 raft_consensus.cc:482] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:04.636103 29457 raft_consensus.cc:504] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.636783 29457 leader_election.cc:288] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:04.637104 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609db8c3f3b245eaa31c82915b61188f" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:04.637265 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609db8c3f3b245eaa31c82915b61188f" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:04.637375 28173 raft_consensus.cc:2454] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:04.637480 28434 raft_consensus.cc:2454] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:04.637874 28240 leader_election.cc:302] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:04.638146 29457 raft_consensus.cc:2781] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:04.638283 29457 raft_consensus.cc:482] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:04.638375 29457 raft_consensus.cc:3037] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.640187 29457 raft_consensus.cc:504] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.641049 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609db8c3f3b245eaa31c82915b61188f" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:04.641111 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609db8c3f3b245eaa31c82915b61188f" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:04.641229 28434 raft_consensus.cc:3037] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.641327 29457 leader_election.cc:288] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:04.641333 28173 raft_consensus.cc:3037] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.643342 28434 raft_consensus.cc:2454] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:04.643723 28240 leader_election.cc:302] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:04.643987 29457 raft_consensus.cc:2781] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:04.644192 28173 raft_consensus.cc:2454] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:04.644644 29457 raft_consensus.cc:686] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:04.644950 29457 consensus_queue.cc:237] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.647639 27902 catalog_manager.cc:5484] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:04.663381 29439 raft_consensus.cc:482] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:04.663530 29439 raft_consensus.cc:504] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.664065 29439 leader_election.cc:288] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:04.664556 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d89642d5364479877c7ba4700fdcee" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:04.664572 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d89642d5364479877c7ba4700fdcee" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:04.664783 28303 raft_consensus.cc:2454] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:04.664801 28173 raft_consensus.cc:2454] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:04.665215 28368 leader_election.cc:302] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:04.665475 29439 raft_consensus.cc:2781] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:04.665588 29439 raft_consensus.cc:482] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:04.665658 29439 raft_consensus.cc:3037] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.667552 29439 raft_consensus.cc:504] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.668056 29439 leader_election.cc:288] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:04.668447 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d89642d5364479877c7ba4700fdcee" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:04.668520 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d89642d5364479877c7ba4700fdcee" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:04.668637 28173 raft_consensus.cc:3037] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.668709 28303 raft_consensus.cc:3037] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.670692 28173 raft_consensus.cc:2454] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:04.671020 28368 leader_election.cc:302] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:04.671270 29439 raft_consensus.cc:2781] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:04.671597 29439 raft_consensus.cc:686] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:04.671867 29439 consensus_queue.cc:237] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:04.674527 27902 catalog_manager.cc:5484] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:04.696118 28303 raft_consensus.cc:2454] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:04.802654 29244 raft_consensus.cc:482] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:04.802815 29244 raft_consensus.cc:504] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.803341 29244 leader_election.cc:288] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:04.803810 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816d34a63b945b08eec771d9e7c2d55" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:04.804001 28042 raft_consensus.cc:2454] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:04.803974 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816d34a63b945b08eec771d9e7c2d55" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:04.804246 28303 raft_consensus.cc:2454] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:04.804486 28110 leader_election.cc:302] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:04.804738 29244 raft_consensus.cc:2781] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:04.804857 29244 raft_consensus.cc:482] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:04.804944 29244 raft_consensus.cc:3037] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.806921 29244 raft_consensus.cc:504] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.807430 29244 leader_election.cc:288] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:04.807842 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816d34a63b945b08eec771d9e7c2d55" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:04.808044 28042 raft_consensus.cc:3037] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.808261 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a816d34a63b945b08eec771d9e7c2d55" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:04.808491 28303 raft_consensus.cc:3037] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.810773 28042 raft_consensus.cc:2454] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:04.811151 28110 leader_election.cc:302] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:04.811170 28303 raft_consensus.cc:2454] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:04.811422 29479 raft_consensus.cc:2781] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:04.811587 29479 raft_consensus.cc:686] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:04.811889 29479 consensus_queue.cc:237] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:04.814471 27902 catalog_manager.cc:5484] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:05.081991 29457 consensus_queue.cc:1035] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:05.088477 29475 consensus_queue.cc:1035] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:05.176152 29481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:05.176517 29481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:05.184131 29481 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240503 14:13:05.232903 29439 consensus_queue.cc:1035] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:05.265149 29512 consensus_queue.cc:1035] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:13:05.327807 29479 consensus_queue.cc:1035] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:05.439385 28043 raft_consensus.cc:1261] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:05.444212 29528 consensus_queue.cc:1035] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240503 14:13:05.514463 29539 mvcc.cc:205] Tried to move back new op lower bound from 7023597917819854848 to 7023597915391385600. Current Snapshot: MvccSnapshot[applied={T|T < 7023597917819854848}]
INSERT report
rows total: 2000
time total: 347.225 ms
time per row: 0.173612 ms
I20240503 14:13:05.682999 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:05.692081 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 95e9ade3fd96457ca25cef44a301644b (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.692435 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 5aae2864fae8454ca494b5e4ba47ac5c (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.692549 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e9ade3fd96457ca25cef44a301644b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.692859 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aae2864fae8454ca494b5e4ba47ac5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.693171 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 95e9ade3fd96457ca25cef44a301644b (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.693114 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 457b7d72f79a4431a9442bea4097b2a8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.693548 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e9ade3fd96457ca25cef44a301644b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.693738 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457b7d72f79a4431a9442bea4097b2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.695056 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 5aae2864fae8454ca494b5e4ba47ac5c (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.695457 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aae2864fae8454ca494b5e4ba47ac5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.698621 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 457b7d72f79a4431a9442bea4097b2a8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.698984 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457b7d72f79a4431a9442bea4097b2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.699898 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 95e9ade3fd96457ca25cef44a301644b (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.700227 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95e9ade3fd96457ca25cef44a301644b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.703186 28152 tablet_service.cc:1467] Processing CreateTablet for tablet 457b7d72f79a4431a9442bea4097b2a8 (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.703426 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 5aae2864fae8454ca494b5e4ba47ac5c (DEFAULT_TABLE table=rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.703604 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 457b7d72f79a4431a9442bea4097b2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.703814 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aae2864fae8454ca494b5e4ba47ac5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.716562 29561 tablet_bootstrap.cc:492] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:05.716867 29559 tablet_bootstrap.cc:492] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:05.716938 29558 tablet_bootstrap.cc:492] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:05.717603 29560 tablet_bootstrap.cc:492] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:05.718863 29561 tablet_bootstrap.cc:654] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.719161 29558 tablet_bootstrap.cc:654] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.719578 29559 tablet_bootstrap.cc:654] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.720887 29560 tablet_bootstrap.cc:654] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.727957 29561 tablet_bootstrap.cc:492] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:05.727964 29559 tablet_bootstrap.cc:492] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:05.728109 29561 ts_tablet_manager.cc:1367] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 14:13:05.728248 29559 ts_tablet_manager.cc:1367] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 14:13:05.728921 29561 raft_consensus.cc:348] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.729162 29561 raft_consensus.cc:374] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.729247 29561 raft_consensus.cc:729] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.729032 29559 raft_consensus.cc:348] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.729487 29559 raft_consensus.cc:374] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.729578 29560 tablet_bootstrap.cc:492] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:05.729622 29559 raft_consensus.cc:729] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.729547 29561 consensus_queue.cc:260] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.729777 29560 ts_tablet_manager.cc:1367] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:13:05.730031 29559 consensus_queue.cc:260] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.730304 29561 ts_tablet_manager.cc:1398] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:13:05.730619 29561 tablet_bootstrap.cc:492] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:05.732252 29561 tablet_bootstrap.cc:654] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.732513 29560 raft_consensus.cc:348] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.732793 29560 raft_consensus.cc:374] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.732889 29560 raft_consensus.cc:729] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.733211 29560 consensus_queue.cc:260] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.734844 29560 ts_tablet_manager.cc:1398] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:05.736172 29558 tablet_bootstrap.cc:492] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:05.736361 29558 ts_tablet_manager.cc:1367] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20240503 14:13:05.736479 29561 tablet_bootstrap.cc:492] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:05.736575 29559 ts_tablet_manager.cc:1398] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:13:05.736636 29561 ts_tablet_manager.cc:1367] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 14:13:05.736933 29559 tablet_bootstrap.cc:492] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:05.737179 29558 raft_consensus.cc:348] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.737381 29558 raft_consensus.cc:374] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.737484 29558 raft_consensus.cc:729] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.737706 29558 consensus_queue.cc:260] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.738533 29558 ts_tablet_manager.cc:1398] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:05.738812 29558 tablet_bootstrap.cc:492] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:05.738822 29559 tablet_bootstrap.cc:654] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.739992 29561 raft_consensus.cc:348] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:05.740235 29561 raft_consensus.cc:374] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.740319 29561 raft_consensus.cc:729] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.740658 29558 tablet_bootstrap.cc:654] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.740617 29561 consensus_queue.cc:260] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:05.741907 29561 ts_tablet_manager.cc:1398] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:05.746707 29558 tablet_bootstrap.cc:492] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:05.746866 29558 ts_tablet_manager.cc:1367] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 14:13:05.747407 29558 raft_consensus.cc:348] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.747573 29558 raft_consensus.cc:374] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.747654 29558 raft_consensus.cc:729] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.747911 29558 consensus_queue.cc:260] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.748375 29558 ts_tablet_manager.cc:1398] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:05.749202 29558 tablet_bootstrap.cc:492] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:05.749729 29559 tablet_bootstrap.cc:492] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:05.749895 29559 ts_tablet_manager.cc:1367] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 14:13:05.750597 29558 tablet_bootstrap.cc:654] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.750694 29559 raft_consensus.cc:348] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.750947 29559 raft_consensus.cc:374] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.751034 29559 raft_consensus.cc:729] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.751333 29559 consensus_queue.cc:260] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.752048 29559 ts_tablet_manager.cc:1398] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:05.752336 29559 tablet_bootstrap.cc:492] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:05.752583 29558 tablet_bootstrap.cc:492] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:05.752717 29558 ts_tablet_manager.cc:1367] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:05.753360 29558 raft_consensus.cc:348] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:05.753546 29558 raft_consensus.cc:374] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.753669 29558 raft_consensus.cc:729] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.753898 29559 tablet_bootstrap.cc:654] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.753882 29558 consensus_queue.cc:260] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:05.754465 29558 ts_tablet_manager.cc:1398] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:05.755751 29559 tablet_bootstrap.cc:492] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:05.755892 29559 ts_tablet_manager.cc:1367] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:05.756652 29559 raft_consensus.cc:348] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:05.756837 29559 raft_consensus.cc:374] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.756930 29559 raft_consensus.cc:729] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.757125 29559 consensus_queue.cc:260] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:05.757560 29559 ts_tablet_manager.cc:1398] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:05.804329 29041 raft_consensus.cc:482] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:05.804579 29041 raft_consensus.cc:504] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.805166 29041 leader_election.cc:288] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:05.805622 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aae2864fae8454ca494b5e4ba47ac5c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:05.805644 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aae2864fae8454ca494b5e4ba47ac5c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:05.805835 28434 raft_consensus.cc:2454] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:05.805851 28304 raft_consensus.cc:2454] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:05.806280 27979 leader_election.cc:302] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:05.806592 29041 raft_consensus.cc:2781] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:05.806676 29041 raft_consensus.cc:482] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:05.806773 29041 raft_consensus.cc:3037] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.808660 29041 raft_consensus.cc:504] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.809207 29041 leader_election.cc:288] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:05.809585 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aae2864fae8454ca494b5e4ba47ac5c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:05.809653 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aae2864fae8454ca494b5e4ba47ac5c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:05.809803 28434 raft_consensus.cc:3037] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.809831 28304 raft_consensus.cc:3037] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.812597 28434 raft_consensus.cc:2454] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:05.812597 28304 raft_consensus.cc:2454] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:05.812973 27979 leader_election.cc:302] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:05.813205 29041 raft_consensus.cc:2781] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:05.813586 29041 raft_consensus.cc:686] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:05.813858 29041 consensus_queue.cc:237] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:05.816538 27898 catalog_manager.cc:5484] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:05.947942 29509 raft_consensus.cc:482] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:05.948160 29509 raft_consensus.cc:504] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.948752 29509 leader_election.cc:288] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:05.949111 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e9ade3fd96457ca25cef44a301644b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:05.949306 28043 raft_consensus.cc:2454] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:05.949323 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e9ade3fd96457ca25cef44a301644b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:05.949561 28174 raft_consensus.cc:2454] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:05.949707 28370 leader_election.cc:302] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:05.949950 29509 raft_consensus.cc:2781] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:05.950088 29509 raft_consensus.cc:482] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:05.950194 29509 raft_consensus.cc:3037] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.952015 29509 raft_consensus.cc:504] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.952533 29509 leader_election.cc:288] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:05.952872 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e9ade3fd96457ca25cef44a301644b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:05.953055 28043 raft_consensus.cc:3037] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.953029 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95e9ade3fd96457ca25cef44a301644b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:05.953217 28174 raft_consensus.cc:3037] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.954972 28043 raft_consensus.cc:2454] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:05.955231 28174 raft_consensus.cc:2454] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:05.955318 28370 leader_election.cc:302] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:05.955647 29509 raft_consensus.cc:2781] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:05.955825 29509 raft_consensus.cc:686] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:05.956115 29509 consensus_queue.cc:237] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:05.959018 27898 catalog_manager.cc:5484] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:06.144308 29509 raft_consensus.cc:482] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:06.144520 29509 raft_consensus.cc:504] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.145092 29509 leader_election.cc:288] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:06.145506 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457b7d72f79a4431a9442bea4097b2a8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:06.145550 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457b7d72f79a4431a9442bea4097b2a8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:06.145701 28043 raft_consensus.cc:2454] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:06.145763 28174 raft_consensus.cc:2454] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:06.146107 28370 leader_election.cc:302] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:06.146390 29509 raft_consensus.cc:2781] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:06.146490 29509 raft_consensus.cc:482] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:06.146566 29509 raft_consensus.cc:3037] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.148286 29509 raft_consensus.cc:504] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.149135 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457b7d72f79a4431a9442bea4097b2a8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:06.149163 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "457b7d72f79a4431a9442bea4097b2a8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:06.149331 28174 raft_consensus.cc:3037] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.149358 28043 raft_consensus.cc:3037] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.150298 29509 leader_election.cc:288] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:06.151170 28174 raft_consensus.cc:2454] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:06.151544 28368 leader_election.cc:302] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:06.151779 29509 raft_consensus.cc:2781] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:06.151926 29509 raft_consensus.cc:686] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:06.152169 28043 raft_consensus.cc:2454] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:06.152196 29509 consensus_queue.cc:237] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.154995 27898 catalog_manager.cc:5484] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:06.248258 29571 consensus_queue.cc:1035] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:06.253868 29571 consensus_queue.cc:1035] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:06.391613 29509 consensus_queue.cc:1035] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:06.396502 29509 consensus_queue.cc:1035] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:06.493307 29575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:06.493758 29575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:06.501670 29575 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240503 14:13:06.602149 29582 consensus_queue.cc:1035] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:06.607537 29509 consensus_queue.cc:1035] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 265.23 ms
time per row: 0.132615 ms
I20240503 14:13:06.914618 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:06.923605 28023 tablet_service.cc:1467] Processing CreateTablet for tablet f133ebe695a74f86be1a37ae661719ec (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.923683 28283 tablet_service.cc:1467] Processing CreateTablet for tablet f133ebe695a74f86be1a37ae661719ec (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.924082 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f133ebe695a74f86be1a37ae661719ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.924135 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f133ebe695a74f86be1a37ae661719ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.924703 28152 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb01c0be7da4fe4b9c7c7160df7debc (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.924769 28153 tablet_service.cc:1467] Processing CreateTablet for tablet f133ebe695a74f86be1a37ae661719ec (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.925134 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb01c0be7da4fe4b9c7c7160df7debc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.926029 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb01c0be7da4fe4b9c7c7160df7debc (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.926472 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb01c0be7da4fe4b9c7c7160df7debc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.926746 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 25eb1d6c445940d6bd8753a88381bb8c (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.927112 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25eb1d6c445940d6bd8753a88381bb8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.930222 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f133ebe695a74f86be1a37ae661719ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.931155 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 25eb1d6c445940d6bd8753a88381bb8c (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.931598 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25eb1d6c445940d6bd8753a88381bb8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.931756 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 25eb1d6c445940d6bd8753a88381bb8c (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.932103 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25eb1d6c445940d6bd8753a88381bb8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.934564 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 1bb01c0be7da4fe4b9c7c7160df7debc (DEFAULT_TABLE table=rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.934918 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb01c0be7da4fe4b9c7c7160df7debc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.945819 29640 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:06.950217 29642 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:06.953478 29642 tablet_bootstrap.cc:654] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.953884 29641 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:06.954630 29643 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:06.954696 29640 tablet_bootstrap.cc:654] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.956276 29643 tablet_bootstrap.cc:654] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.956851 29642 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:06.957038 29642 ts_tablet_manager.cc:1367] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:13:06.957237 29640 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:06.957515 29640 ts_tablet_manager.cc:1367] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240503 14:13:06.958142 29641 tablet_bootstrap.cc:654] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.958052 29642 raft_consensus.cc:348] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.958395 29642 raft_consensus.cc:374] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.958550 29642 raft_consensus.cc:729] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.958619 29640 raft_consensus.cc:348] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.958948 29640 raft_consensus.cc:374] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.959090 29640 raft_consensus.cc:729] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.959225 29642 consensus_queue.cc:260] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.960011 29640 consensus_queue.cc:260] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.960834 29640 ts_tablet_manager.cc:1398] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:06.961364 29640 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:06.962230 29642 ts_tablet_manager.cc:1398] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:06.962927 29640 tablet_bootstrap.cc:654] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.963208 29642 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:06.966542 29641 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:06.966732 29641 ts_tablet_manager.cc:1367] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20240503 14:13:06.967554 29641 raft_consensus.cc:348] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.967818 29641 raft_consensus.cc:374] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.967909 29642 tablet_bootstrap.cc:654] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.967957 29641 raft_consensus.cc:729] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.968257 29641 consensus_queue.cc:260] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.968927 29641 ts_tablet_manager.cc:1398] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:06.969344 29641 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:06.972216 29641 tablet_bootstrap.cc:654] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.974187 29640 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:06.974288 29642 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:06.974406 29640 ts_tablet_manager.cc:1367] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20240503 14:13:06.974557 29642 ts_tablet_manager.cc:1367] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:13:06.975196 29640 raft_consensus.cc:348] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.975446 29640 raft_consensus.cc:374] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.975560 29640 raft_consensus.cc:729] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.975857 29640 consensus_queue.cc:260] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.975858 29642 raft_consensus.cc:348] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.976624 29640 ts_tablet_manager.cc:1398] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:06.977092 29640 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:06.977560 29642 raft_consensus.cc:374] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.977638 29641 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:06.977711 29642 raft_consensus.cc:729] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.977779 29641 ts_tablet_manager.cc:1367] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240503 14:13:06.978044 29642 consensus_queue.cc:260] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.978542 29641 raft_consensus.cc:348] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.978745 29641 raft_consensus.cc:374] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.978826 29641 raft_consensus.cc:729] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.979005 29640 tablet_bootstrap.cc:654] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.979064 29641 consensus_queue.cc:260] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:06.979700 29641 ts_tablet_manager.cc:1398] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:06.980180 29642 ts_tablet_manager.cc:1398] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:06.981235 29640 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:06.981389 29640 ts_tablet_manager.cc:1367] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:06.982139 29640 raft_consensus.cc:348] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.982357 29640 raft_consensus.cc:374] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.982378 29643 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:06.982458 29640 raft_consensus.cc:729] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.982537 29643 ts_tablet_manager.cc:1367] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.008s
I20240503 14:13:06.982685 29640 consensus_queue.cc:260] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.983345 29643 raft_consensus.cc:348] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.983529 29643 raft_consensus.cc:374] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.983611 29643 raft_consensus.cc:729] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.983841 29643 consensus_queue.cc:260] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.983990 29640 ts_tablet_manager.cc:1398] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:06.984397 29643 ts_tablet_manager.cc:1398] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:06.984648 29643 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:06.986392 29643 tablet_bootstrap.cc:654] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.988126 29643 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:06.988632 29643 ts_tablet_manager.cc:1367] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:06.989567 29643 raft_consensus.cc:348] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.989781 29643 raft_consensus.cc:374] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.989868 29643 raft_consensus.cc:729] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.990123 29643 consensus_queue.cc:260] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:06.990666 29643 ts_tablet_manager.cc:1398] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:07.045260 29457 raft_consensus.cc:482] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:07.045451 29457 raft_consensus.cc:504] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.046118 29457 leader_election.cc:288] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:07.046291 29457 raft_consensus.cc:482] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:07.046427 29457 raft_consensus.cc:504] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.046708 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f133ebe695a74f86be1a37ae661719ec" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:07.047032 29457 leader_election.cc:288] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:07.047049 28042 raft_consensus.cc:2454] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:07.047315 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f133ebe695a74f86be1a37ae661719ec" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:07.047559 28173 raft_consensus.cc:2454] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:07.047605 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb01c0be7da4fe4b9c7c7160df7debc" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:07.047767 28174 raft_consensus.cc:2454] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:07.048056 28238 leader_election.cc:302] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:07.048429 29653 raft_consensus.cc:2781] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:07.048434 28238 leader_election.cc:302] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:07.048542 29653 raft_consensus.cc:482] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:07.048692 29653 raft_consensus.cc:3037] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.048883 29457 raft_consensus.cc:2781] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:07.048980 29457 raft_consensus.cc:482] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:07.048939 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb01c0be7da4fe4b9c7c7160df7debc" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:07.049090 29457 raft_consensus.cc:3037] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.049129 28434 raft_consensus.cc:2454] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:07.051801 29653 raft_consensus.cc:504] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.051740 29457 raft_consensus.cc:504] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.052453 29653 leader_election.cc:288] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:07.052786 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f133ebe695a74f86be1a37ae661719ec" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:07.052855 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f133ebe695a74f86be1a37ae661719ec" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:07.052994 28042 raft_consensus.cc:3037] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.053030 28174 raft_consensus.cc:3037] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.053205 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb01c0be7da4fe4b9c7c7160df7debc" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:07.053386 28173 raft_consensus.cc:3037] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.053449 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb01c0be7da4fe4b9c7c7160df7debc" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:07.053612 28434 raft_consensus.cc:3037] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.055981 28174 raft_consensus.cc:2454] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:07.056015 28042 raft_consensus.cc:2454] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:07.056401 28238 leader_election.cc:302] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:07.056461 28434 raft_consensus.cc:2454] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:07.056623 29457 leader_election.cc:288] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:07.056711 28173 raft_consensus.cc:2454] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:07.056836 29457 raft_consensus.cc:2781] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:07.056867 28240 leader_election.cc:302] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:07.056993 29457 raft_consensus.cc:686] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:07.057195 29653 raft_consensus.cc:2781] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:07.057307 29457 consensus_queue.cc:237] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.057490 29653 raft_consensus.cc:686] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:07.057895 29653 consensus_queue.cc:237] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.060595 27900 catalog_manager.cc:5484] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:07.065171 27900 catalog_manager.cc:5484] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:07.107949 29634 raft_consensus.cc:482] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:07.108107 29634 raft_consensus.cc:504] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.108785 29634 leader_election.cc:288] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:07.109272 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eb1d6c445940d6bd8753a88381bb8c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:07.109300 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eb1d6c445940d6bd8753a88381bb8c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:07.109531 28434 raft_consensus.cc:2454] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:07.109534 28304 raft_consensus.cc:2454] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:07.109938 27979 leader_election.cc:302] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:07.110162 29634 raft_consensus.cc:2781] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:07.110285 29634 raft_consensus.cc:482] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:07.110376 29634 raft_consensus.cc:3037] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.112296 29634 raft_consensus.cc:504] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.112767 29634 leader_election.cc:288] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:07.113162 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eb1d6c445940d6bd8753a88381bb8c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:07.113229 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eb1d6c445940d6bd8753a88381bb8c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:07.113337 28304 raft_consensus.cc:3037] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.113431 28434 raft_consensus.cc:3037] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.115170 28304 raft_consensus.cc:2454] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:07.115289 28434 raft_consensus.cc:2454] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:07.115509 27979 leader_election.cc:302] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:07.115782 29634 raft_consensus.cc:2781] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:07.115939 29634 raft_consensus.cc:686] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:07.116197 29634 consensus_queue.cc:237] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.118944 27900 catalog_manager.cc:5484] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
W20240503 14:13:07.462281 29659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:07.462610 29659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:07.470504 29659 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 14:13:07.508116 29634 consensus_queue.cc:1035] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.520772 29632 consensus_queue.cc:1035] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.583324 29653 consensus_queue.cc:1035] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.590354 29653 consensus_queue.cc:1035] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.669819 28174 raft_consensus.cc:1261] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:07.670589 29688 consensus_queue.cc:1035] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.684705 28043 raft_consensus.cc:1261] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:07.722208 29711 consensus_queue.cc:1035] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:07.749958 29673 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.20.240.67:43889, user_credentials={real_user=slave}} blocked reactor thread for 44707us
W20240503 14:13:07.752575 28238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 65940.5us
INSERT report
rows total: 2000
time total: 259.372 ms
time per row: 0.129686 ms
I20240503 14:13:07.887523 27897 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:07.895871 28282 tablet_service.cc:1467] Processing CreateTablet for tablet b4a09a046a5043a59c50c359304caa85 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.896067 28023 tablet_service.cc:1467] Processing CreateTablet for tablet b4a09a046a5043a59c50c359304caa85 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.896420 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a09a046a5043a59c50c359304caa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.896489 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a09a046a5043a59c50c359304caa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.897708 28152 tablet_service.cc:1467] Processing CreateTablet for tablet b4a09a046a5043a59c50c359304caa85 (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.898069 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4a09a046a5043a59c50c359304caa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.898264 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 99896ce4c4ec4384a84e4b07d3a19b2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.898674 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99896ce4c4ec4384a84e4b07d3a19b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.899348 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 99896ce4c4ec4384a84e4b07d3a19b2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.899752 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99896ce4c4ec4384a84e4b07d3a19b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.902494 28413 tablet_service.cc:1467] Processing CreateTablet for tablet e90f07adac794442be03689f97bfe60a (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.902590 28283 tablet_service.cc:1467] Processing CreateTablet for tablet e90f07adac794442be03689f97bfe60a (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.902832 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90f07adac794442be03689f97bfe60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.902961 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90f07adac794442be03689f97bfe60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.907141 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 99896ce4c4ec4384a84e4b07d3a19b2d (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.907387 28021 tablet_service.cc:1467] Processing CreateTablet for tablet e90f07adac794442be03689f97bfe60a (DEFAULT_TABLE table=rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.907534 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99896ce4c4ec4384a84e4b07d3a19b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.907709 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e90f07adac794442be03689f97bfe60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.922937 29724 tablet_bootstrap.cc:492] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:07.925447 29724 tablet_bootstrap.cc:654] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.925521 29725 tablet_bootstrap.cc:492] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:07.926625 29726 tablet_bootstrap.cc:492] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:07.927541 29725 tablet_bootstrap.cc:654] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.928309 29726 tablet_bootstrap.cc:654] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.928992 29723 tablet_bootstrap.cc:492] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:07.929338 29724 tablet_bootstrap.cc:492] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:07.929531 29724 ts_tablet_manager.cc:1367] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:13:07.930478 29724 raft_consensus.cc:348] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.930687 29723 tablet_bootstrap.cc:654] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.930867 29724 raft_consensus.cc:374] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.930984 29724 raft_consensus.cc:729] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.931231 29724 consensus_queue.cc:260] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.931905 29724 ts_tablet_manager.cc:1398] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:07.932997 29724 tablet_bootstrap.cc:492] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:07.934935 29724 tablet_bootstrap.cc:654] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.938310 29726 tablet_bootstrap.cc:492] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:07.938571 29726 ts_tablet_manager.cc:1367] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240503 14:13:07.939534 29726 raft_consensus.cc:348] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.939795 29726 raft_consensus.cc:374] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.939911 29726 raft_consensus.cc:729] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.940150 29723 tablet_bootstrap.cc:492] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:07.940299 29723 ts_tablet_manager.cc:1367] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240503 14:13:07.940198 29726 consensus_queue.cc:260] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.941227 29723 raft_consensus.cc:348] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.941864 29723 raft_consensus.cc:374] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.942061 29723 raft_consensus.cc:729] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.942418 29723 consensus_queue.cc:260] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.942622 29726 ts_tablet_manager.cc:1398] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:07.943197 29723 ts_tablet_manager.cc:1398] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:07.943537 29723 tablet_bootstrap.cc:492] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:07.944140 29726 tablet_bootstrap.cc:492] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:07.945184 29723 tablet_bootstrap.cc:654] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.945987 29726 tablet_bootstrap.cc:654] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.946847 29724 tablet_bootstrap.cc:492] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:07.947050 29724 ts_tablet_manager.cc:1367] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240503 14:13:07.947793 29724 raft_consensus.cc:348] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.948002 29724 raft_consensus.cc:374] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.948089 29724 raft_consensus.cc:729] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.948357 29724 consensus_queue.cc:260] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.949342 29724 ts_tablet_manager.cc:1398] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:07.949779 29724 tablet_bootstrap.cc:492] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:07.950390 29725 tablet_bootstrap.cc:492] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:07.950559 29725 ts_tablet_manager.cc:1367] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.012s
I20240503 14:13:07.951184 29724 tablet_bootstrap.cc:654] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.951603 29725 raft_consensus.cc:348] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.952355 29725 raft_consensus.cc:374] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.952543 29725 raft_consensus.cc:729] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.952912 29725 consensus_queue.cc:260] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.953338 29724 tablet_bootstrap.cc:492] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:07.954010 29723 tablet_bootstrap.cc:492] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:07.954166 29723 ts_tablet_manager.cc:1367] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240503 14:13:07.954805 29724 ts_tablet_manager.cc:1367] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:07.954887 29723 raft_consensus.cc:348] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.955137 29723 raft_consensus.cc:374] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.955229 29723 raft_consensus.cc:729] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.955500 29723 consensus_queue.cc:260] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.955574 29724 raft_consensus.cc:348] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.955761 29724 raft_consensus.cc:374] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.955871 29724 raft_consensus.cc:729] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.956173 29723 ts_tablet_manager.cc:1398] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:07.956481 29724 consensus_queue.cc:260] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:07.957911 29725 ts_tablet_manager.cc:1398] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:07.958590 29724 ts_tablet_manager.cc:1398] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:07.958884 29725 tablet_bootstrap.cc:492] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:07.960757 29726 tablet_bootstrap.cc:492] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:07.961092 29725 tablet_bootstrap.cc:654] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.961117 29726 ts_tablet_manager.cc:1367] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240503 14:13:07.961905 29726 raft_consensus.cc:348] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.962280 29726 raft_consensus.cc:374] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.962414 29726 raft_consensus.cc:729] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.962829 29726 consensus_queue.cc:260] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.964092 29726 ts_tablet_manager.cc:1398] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:07.965261 29725 tablet_bootstrap.cc:492] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:07.965394 29725 ts_tablet_manager.cc:1367] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 14:13:07.966140 29725 raft_consensus.cc:348] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.966365 29725 raft_consensus.cc:374] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.966446 29725 raft_consensus.cc:729] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.966701 29725 consensus_queue.cc:260] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:07.967285 29725 ts_tablet_manager.cc:1398] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:08.025677 29715 raft_consensus.cc:482] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:08.025872 29715 raft_consensus.cc:504] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:08.026585 29715 leader_election.cc:288] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:08.027146 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a09a046a5043a59c50c359304caa85" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:08.027156 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a09a046a5043a59c50c359304caa85" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:08.027367 28304 raft_consensus.cc:2454] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:08.027386 28173 raft_consensus.cc:2454] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:08.027792 27977 leader_election.cc:302] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:08.028043 29715 raft_consensus.cc:2781] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:08.028156 29715 raft_consensus.cc:482] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:08.028249 29715 raft_consensus.cc:3037] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.030282 29715 raft_consensus.cc:504] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:08.030900 29715 leader_election.cc:288] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:08.031342 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a09a046a5043a59c50c359304caa85" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:08.031426 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4a09a046a5043a59c50c359304caa85" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:08.031560 28304 raft_consensus.cc:3037] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.031620 28173 raft_consensus.cc:3037] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.034513 28304 raft_consensus.cc:2454] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:08.034499 28173 raft_consensus.cc:2454] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:08.034911 27977 leader_election.cc:302] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:08.035238 29715 raft_consensus.cc:2781] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:08.035411 29715 raft_consensus.cc:686] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:08.035692 29715 consensus_queue.cc:237] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:08.038782 27899 catalog_manager.cc:5484] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:08.121629 29715 raft_consensus.cc:482] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:08.121816 29715 raft_consensus.cc:504] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:08.122473 29715 leader_election.cc:288] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:08.122851 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99896ce4c4ec4384a84e4b07d3a19b2d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:08.122916 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99896ce4c4ec4384a84e4b07d3a19b2d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:08.123073 28433 raft_consensus.cc:2454] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:08.123176 28173 raft_consensus.cc:2454] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:08.123523 27979 leader_election.cc:302] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:08.123832 29715 raft_consensus.cc:2781] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:08.123930 29715 raft_consensus.cc:482] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:08.124011 29715 raft_consensus.cc:3037] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.125841 29715 raft_consensus.cc:504] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:08.126377 29715 leader_election.cc:288] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:08.126690 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99896ce4c4ec4384a84e4b07d3a19b2d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:08.126772 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99896ce4c4ec4384a84e4b07d3a19b2d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:08.126875 28433 raft_consensus.cc:3037] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.126956 28173 raft_consensus.cc:3037] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.129137 28433 raft_consensus.cc:2454] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:08.129220 28173 raft_consensus.cc:2454] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:08.129562 27979 leader_election.cc:302] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:08.129873 29715 raft_consensus.cc:2781] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:08.130043 29715 raft_consensus.cc:686] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:08.130301 29715 consensus_queue.cc:237] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:08.132843 27899 catalog_manager.cc:5484] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:08.371307 29704 raft_consensus.cc:482] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:08.371502 29704 raft_consensus.cc:504] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:08.372112 29704 leader_election.cc:288] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:08.372709 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f07adac794442be03689f97bfe60a" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:08.372738 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f07adac794442be03689f97bfe60a" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:08.372958 28434 raft_consensus.cc:2454] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:08.372958 28043 raft_consensus.cc:2454] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:08.373365 28240 leader_election.cc:302] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:08.373603 29704 raft_consensus.cc:2781] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:08.373724 29704 raft_consensus.cc:482] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:08.373821 29704 raft_consensus.cc:3037] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.375860 29704 raft_consensus.cc:504] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:08.376353 29704 leader_election.cc:288] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:08.376782 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f07adac794442be03689f97bfe60a" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:08.376888 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e90f07adac794442be03689f97bfe60a" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:08.376978 28043 raft_consensus.cc:3037] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.377094 28434 raft_consensus.cc:3037] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.378784 28043 raft_consensus.cc:2454] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:08.378878 28434 raft_consensus.cc:2454] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:08.379133 28240 leader_election.cc:302] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:08.379379 29704 raft_consensus.cc:2781] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:08.379513 29704 raft_consensus.cc:686] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:08.379830 29704 consensus_queue.cc:237] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:08.382989 27899 catalog_manager.cc:5484] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:08.485829 29715 consensus_queue.cc:1035] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.508934 29715 consensus_queue.cc:1035] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.591807 29715 consensus_queue.cc:1035] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.596673 29715 consensus_queue.cc:1035] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:08.837309 29741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:08.837600 29741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:08.844954 29741 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 14:13:08.908982 29704 consensus_queue.cc:1035] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.949797 29704 consensus_queue.cc:1035] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 255.734 ms
time per row: 0.127867 ms
I20240503 14:13:09.256556 27902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:09.266048 28152 tablet_service.cc:1467] Processing CreateTablet for tablet 05a67063845044eea2d56c34cdfd4ab1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.266553 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a67063845044eea2d56c34cdfd4ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.266716 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 05a67063845044eea2d56c34cdfd4ab1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.267086 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a67063845044eea2d56c34cdfd4ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.271518 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 05a67063845044eea2d56c34cdfd4ab1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.271631 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 69d4d86cb68646bc9141c1e6df964561 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.271929 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a67063845044eea2d56c34cdfd4ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.272140 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d4d86cb68646bc9141c1e6df964561. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.275255 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 529c250699cb4e09bae6129d138e37c2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.275314 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 69d4d86cb68646bc9141c1e6df964561 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.275655 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529c250699cb4e09bae6129d138e37c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.275800 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d4d86cb68646bc9141c1e6df964561. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.278802 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 529c250699cb4e09bae6129d138e37c2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.279255 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529c250699cb4e09bae6129d138e37c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.280984 29805 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:09.282146 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 69d4d86cb68646bc9141c1e6df964561 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.282577 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d4d86cb68646bc9141c1e6df964561. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.283610 29805 tablet_bootstrap.cc:654] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.286760 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 529c250699cb4e09bae6129d138e37c2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.287269 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529c250699cb4e09bae6129d138e37c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.287729 29806 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:09.289139 29806 tablet_bootstrap.cc:654] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.291610 29806 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:09.291751 29806 ts_tablet_manager.cc:1367] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:09.292368 29806 raft_consensus.cc:348] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.292532 29806 raft_consensus.cc:374] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.292632 29806 raft_consensus.cc:729] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.292860 29806 consensus_queue.cc:260] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.293458 29806 ts_tablet_manager.cc:1398] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:09.293835 29806 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:09.295063 29806 tablet_bootstrap.cc:654] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.297051 29806 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:09.297263 29806 ts_tablet_manager.cc:1367] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:09.298017 29806 raft_consensus.cc:348] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.298209 29806 raft_consensus.cc:374] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.298313 29806 raft_consensus.cc:729] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.298539 29806 consensus_queue.cc:260] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.299094 29806 ts_tablet_manager.cc:1398] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:09.299423 29806 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:09.300664 29806 tablet_bootstrap.cc:654] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.309669 29806 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:09.309863 29806 ts_tablet_manager.cc:1367] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 14:13:09.310689 29806 raft_consensus.cc:348] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.310933 29806 raft_consensus.cc:374] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.311069 29806 raft_consensus.cc:729] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.311347 29806 consensus_queue.cc:260] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.311923 29806 ts_tablet_manager.cc:1398] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:09.315263 29810 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:09.315510 29812 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:09.317251 29805 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:09.317364 29810 tablet_bootstrap.cc:654] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.317406 29805 ts_tablet_manager.cc:1367] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.025s
I20240503 14:13:09.317987 29805 raft_consensus.cc:348] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.318149 29805 raft_consensus.cc:374] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.318961 29810 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:09.319092 29810 ts_tablet_manager.cc:1367] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:09.319343 29812 tablet_bootstrap.cc:654] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.319731 29810 raft_consensus.cc:348] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.319895 29810 raft_consensus.cc:374] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.319984 29810 raft_consensus.cc:729] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.320113 29805 raft_consensus.cc:729] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.320312 29810 consensus_queue.cc:260] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.320389 29805 consensus_queue.cc:260] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.320887 29810 ts_tablet_manager.cc:1398] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:09.321123 29805 ts_tablet_manager.cc:1398] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:09.321270 29810 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:09.321475 29805 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:09.323173 29805 tablet_bootstrap.cc:654] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.323654 29810 tablet_bootstrap.cc:654] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.326196 29812 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:09.326351 29812 ts_tablet_manager.cc:1367] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240503 14:13:09.327126 29812 raft_consensus.cc:348] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.327395 29812 raft_consensus.cc:374] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.327546 29812 raft_consensus.cc:729] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.327865 29812 consensus_queue.cc:260] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.328513 29812 ts_tablet_manager.cc:1398] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:09.333546 29810 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:09.333791 29810 ts_tablet_manager.cc:1367] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20240503 14:13:09.334317 29805 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:09.334510 29805 ts_tablet_manager.cc:1367] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240503 14:13:09.334518 29810 raft_consensus.cc:348] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.334767 29810 raft_consensus.cc:374] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.334885 29810 raft_consensus.cc:729] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.335137 29810 consensus_queue.cc:260] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.335258 29805 raft_consensus.cc:348] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.335458 29805 raft_consensus.cc:374] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.335542 29805 raft_consensus.cc:729] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.335677 29810 ts_tablet_manager.cc:1398] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:09.335776 29805 consensus_queue.cc:260] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.336284 29805 ts_tablet_manager.cc:1398] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:09.336694 29805 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:09.338201 29805 tablet_bootstrap.cc:654] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.340207 29805 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:09.340318 29805 ts_tablet_manager.cc:1367] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:09.341020 29805 raft_consensus.cc:348] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.341207 29805 raft_consensus.cc:374] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.341347 29805 raft_consensus.cc:729] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.341565 29805 consensus_queue.cc:260] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.342036 29805 ts_tablet_manager.cc:1398] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:09.369319 29790 raft_consensus.cc:482] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:09.369493 29790 raft_consensus.cc:504] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.370262 29790 leader_election.cc:288] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:09.370653 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529c250699cb4e09bae6129d138e37c2" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:09.370836 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529c250699cb4e09bae6129d138e37c2" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:09.370893 28303 raft_consensus.cc:2454] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:09.371063 28173 raft_consensus.cc:2454] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:09.371563 27977 leader_election.cc:302] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:09.371858 29634 raft_consensus.cc:2781] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:09.371960 29634 raft_consensus.cc:482] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:09.372082 29634 raft_consensus.cc:3037] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.374089 29634 raft_consensus.cc:504] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.374680 29634 leader_election.cc:288] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:09.375066 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529c250699cb4e09bae6129d138e37c2" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:09.375136 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "529c250699cb4e09bae6129d138e37c2" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:09.375241 28303 raft_consensus.cc:3037] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.375309 28173 raft_consensus.cc:3037] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.377288 28303 raft_consensus.cc:2454] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:09.377334 28173 raft_consensus.cc:2454] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:09.377661 27979 leader_election.cc:302] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:09.377900 29634 raft_consensus.cc:2781] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:09.378039 29634 raft_consensus.cc:686] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:09.378351 29737 raft_consensus.cc:482] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:09.378502 29737 raft_consensus.cc:504] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.379079 29737 leader_election.cc:288] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:09.379500 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:09.379552 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:09.379499 29634 consensus_queue.cc:237] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:09.379730 28303 raft_consensus.cc:2454] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:09.379756 28173 raft_consensus.cc:2454] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:09.380120 28368 leader_election.cc:302] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:09.380419 29737 raft_consensus.cc:2781] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:09.380537 29737 raft_consensus.cc:482] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:09.380616 29737 raft_consensus.cc:3037] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.382411 29737 raft_consensus.cc:504] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.382905 29737 leader_election.cc:288] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:09.383366 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:09.383366 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:09.383566 28173 raft_consensus.cc:3037] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.383565 28303 raft_consensus.cc:3037] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.386401 28303 raft_consensus.cc:2454] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:09.386401 28173 raft_consensus.cc:2454] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:09.386744 28370 leader_election.cc:302] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:09.387017 29737 raft_consensus.cc:2781] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:09.387472 29737 raft_consensus.cc:686] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:09.387796 29737 consensus_queue.cc:237] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:09.391305 27899 catalog_manager.cc:5484] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:09.394217 27902 catalog_manager.cc:5484] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:09.475086 29737 raft_consensus.cc:482] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:09.475270 29737 raft_consensus.cc:504] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.475894 29737 leader_election.cc:288] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:09.476383 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d4d86cb68646bc9141c1e6df964561" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:09.476512 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d4d86cb68646bc9141c1e6df964561" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:09.476660 28173 raft_consensus.cc:2454] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:09.476722 28303 raft_consensus.cc:2454] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:09.477070 28370 leader_election.cc:302] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:09.477434 29737 raft_consensus.cc:2781] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:09.477526 29737 raft_consensus.cc:482] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:09.477634 29737 raft_consensus.cc:3037] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.479707 29737 raft_consensus.cc:504] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.480293 29737 leader_election.cc:288] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:09.480605 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d4d86cb68646bc9141c1e6df964561" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:09.480808 28173 raft_consensus.cc:3037] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.480792 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d4d86cb68646bc9141c1e6df964561" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:09.481014 28303 raft_consensus.cc:3037] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.483672 28173 raft_consensus.cc:2454] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:09.483716 28303 raft_consensus.cc:2454] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:09.484074 28368 leader_election.cc:302] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:09.484472 29737 raft_consensus.cc:2781] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:09.484679 29737 raft_consensus.cc:686] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:09.485062 29737 consensus_queue.cc:237] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:09.492322 27902 catalog_manager.cc:5484] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:09.813632 29737 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.819568 29737 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.820127 29634 consensus_queue.cc:1035] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.832690 29790 consensus_queue.cc:1035] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:09.877203 29822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:09.877518 29822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:09.885054 29822 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240503 14:13:09.932870 29737 consensus_queue.cc:1035] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.939159 29737 consensus_queue.cc:1035] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 229.373 ms
time per row: 0.114686 ms
I20240503 14:13:10.266197 27902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:10.274416 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4c8fcd60bc42e496d9ca2067a43115 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.274895 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4c8fcd60bc42e496d9ca2067a43115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.274791 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4c8fcd60bc42e496d9ca2067a43115 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.275199 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4c8fcd60bc42e496d9ca2067a43115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.275532 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4c8fcd60bc42e496d9ca2067a43115 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.276010 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4c8fcd60bc42e496d9ca2067a43115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.278458 28152 tablet_service.cc:1467] Processing CreateTablet for tablet 1598708982124ee69f177945753d3703 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.278828 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1598708982124ee69f177945753d3703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.281684 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 6bff51b843af4f488cde34745619d3c8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.282167 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bff51b843af4f488cde34745619d3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.283038 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 6bff51b843af4f488cde34745619d3c8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.283573 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bff51b843af4f488cde34745619d3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.286911 29886 tablet_bootstrap.cc:492] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:10.286885 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 6bff51b843af4f488cde34745619d3c8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.287303 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bff51b843af4f488cde34745619d3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.289287 29886 tablet_bootstrap.cc:654] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.290627 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 1598708982124ee69f177945753d3703 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.291018 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1598708982124ee69f177945753d3703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.287303 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 1598708982124ee69f177945753d3703 (DEFAULT_TABLE table=rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.294941 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1598708982124ee69f177945753d3703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.295250 29889 tablet_bootstrap.cc:492] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:10.297649 29889 tablet_bootstrap.cc:654] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.300175 29890 tablet_bootstrap.cc:492] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:10.300449 29886 tablet_bootstrap.cc:492] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:10.300628 29886 ts_tablet_manager.cc:1367] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20240503 14:13:10.301448 29886 raft_consensus.cc:348] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.301714 29886 raft_consensus.cc:374] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.301862 29886 raft_consensus.cc:729] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.302497 29886 consensus_queue.cc:260] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.301836 29890 tablet_bootstrap.cc:654] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.305042 29886 ts_tablet_manager.cc:1398] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:10.305368 29886 tablet_bootstrap.cc:492] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:10.306039 29887 tablet_bootstrap.cc:492] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:10.307497 29886 tablet_bootstrap.cc:654] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.309163 29887 tablet_bootstrap.cc:654] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.310220 29889 tablet_bootstrap.cc:492] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:10.310429 29889 ts_tablet_manager.cc:1367] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240503 14:13:10.311484 29889 raft_consensus.cc:348] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.311758 29889 raft_consensus.cc:374] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.311898 29889 raft_consensus.cc:729] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.312188 29889 consensus_queue.cc:260] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.312400 29886 tablet_bootstrap.cc:492] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:10.312584 29886 ts_tablet_manager.cc:1367] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.003s
I20240503 14:13:10.312922 29889 ts_tablet_manager.cc:1398] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:10.313278 29889 tablet_bootstrap.cc:492] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:10.313338 29886 raft_consensus.cc:348] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.313593 29886 raft_consensus.cc:374] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.313737 29886 raft_consensus.cc:729] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.314031 29886 consensus_queue.cc:260] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.314741 29886 ts_tablet_manager.cc:1398] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:10.315332 29886 tablet_bootstrap.cc:492] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:10.315975 29890 tablet_bootstrap.cc:492] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:10.316169 29890 ts_tablet_manager.cc:1367] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.007s
I20240503 14:13:10.317008 29890 raft_consensus.cc:348] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.317282 29890 raft_consensus.cc:374] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.317415 29890 raft_consensus.cc:729] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.317701 29890 consensus_queue.cc:260] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.318867 29890 ts_tablet_manager.cc:1398] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:10.320921 29886 tablet_bootstrap.cc:654] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.321210 29889 tablet_bootstrap.cc:654] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.322190 29887 tablet_bootstrap.cc:492] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:10.322443 29887 ts_tablet_manager.cc:1367] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20240503 14:13:10.323853 29887 raft_consensus.cc:348] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.324121 29887 raft_consensus.cc:374] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.324258 29887 raft_consensus.cc:729] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.324324 29886 tablet_bootstrap.cc:492] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:10.324524 29886 ts_tablet_manager.cc:1367] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240503 14:13:10.324553 29887 consensus_queue.cc:260] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.325412 29886 raft_consensus.cc:348] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.325656 29886 raft_consensus.cc:374] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.325672 29887 ts_tablet_manager.cc:1398] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:10.325943 29887 tablet_bootstrap.cc:492] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:10.326009 29886 raft_consensus.cc:729] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.326557 29886 consensus_queue.cc:260] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.327337 29886 ts_tablet_manager.cc:1398] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:10.327886 29887 tablet_bootstrap.cc:654] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.330904 29889 tablet_bootstrap.cc:492] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:10.331076 29889 ts_tablet_manager.cc:1367] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240503 14:13:10.331354 29887 tablet_bootstrap.cc:492] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:10.331503 29887 ts_tablet_manager.cc:1367] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 14:13:10.331977 29889 raft_consensus.cc:348] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.332139 29889 raft_consensus.cc:374] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.332095 29887 raft_consensus.cc:348] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.332211 29889 raft_consensus.cc:729] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.332329 29887 raft_consensus.cc:374] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.332412 29887 raft_consensus.cc:729] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.332434 29889 consensus_queue.cc:260] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.332652 29887 consensus_queue.cc:260] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.332955 29889 ts_tablet_manager.cc:1398] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:10.333174 29887 ts_tablet_manager.cc:1398] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:10.333364 29889 tablet_bootstrap.cc:492] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:10.334695 29889 tablet_bootstrap.cc:654] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.336731 29889 tablet_bootstrap.cc:492] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:10.336915 29889 ts_tablet_manager.cc:1367] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:10.337478 29889 raft_consensus.cc:348] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.337680 29889 raft_consensus.cc:374] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.337780 29889 raft_consensus.cc:729] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.337975 29889 consensus_queue.cc:260] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.338480 29889 ts_tablet_manager.cc:1398] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:10.523200 29874 raft_consensus.cc:482] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:10.523392 29874 raft_consensus.cc:504] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.524108 29874 leader_election.cc:288] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:10.524541 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bff51b843af4f488cde34745619d3c8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:10.524770 28303 raft_consensus.cc:2454] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:10.524900 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bff51b843af4f488cde34745619d3c8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:10.525179 28043 raft_consensus.cc:2454] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:10.525213 28370 leader_election.cc:302] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:10.525554 29874 raft_consensus.cc:2781] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:10.525683 29874 raft_consensus.cc:482] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:10.525794 29874 raft_consensus.cc:3037] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.527673 29874 raft_consensus.cc:504] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.528198 29874 leader_election.cc:288] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:10.528637 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bff51b843af4f488cde34745619d3c8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:10.528652 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bff51b843af4f488cde34745619d3c8" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:10.528820 28043 raft_consensus.cc:3037] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.528826 28304 raft_consensus.cc:3037] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.530653 28304 raft_consensus.cc:2454] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:10.530781 28043 raft_consensus.cc:2454] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:10.531052 28370 leader_election.cc:302] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:10.531335 29874 raft_consensus.cc:2781] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:10.531495 29874 raft_consensus.cc:686] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:10.531752 29874 consensus_queue.cc:237] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:10.534793 27897 catalog_manager.cc:5484] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:10.573596 29855 raft_consensus.cc:482] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:10.573789 29855 raft_consensus.cc:504] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.574505 29855 leader_election.cc:288] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:10.574911 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1598708982124ee69f177945753d3703" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:10.575064 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1598708982124ee69f177945753d3703" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:10.575189 28043 raft_consensus.cc:2454] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:10.575270 28433 raft_consensus.cc:2454] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:10.575663 28110 leader_election.cc:302] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:10.576005 29855 raft_consensus.cc:2781] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:10.576133 29855 raft_consensus.cc:482] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:10.576233 29855 raft_consensus.cc:3037] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.578100 29855 raft_consensus.cc:504] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.578555 29855 leader_election.cc:288] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:10.578907 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1598708982124ee69f177945753d3703" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:10.579102 28043 raft_consensus.cc:3037] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.579049 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1598708982124ee69f177945753d3703" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:10.579231 28433 raft_consensus.cc:3037] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.581141 28043 raft_consensus.cc:2454] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:10.581226 28433 raft_consensus.cc:2454] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:10.581472 28110 leader_election.cc:302] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:10.581723 29855 raft_consensus.cc:2781] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:10.582114 29855 raft_consensus.cc:686] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:10.582435 29855 consensus_queue.cc:237] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.585285 27897 catalog_manager.cc:5484] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:10.625943 29855 raft_consensus.cc:482] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:10.626119 29855 raft_consensus.cc:504] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.626662 29855 leader_election.cc:288] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:10.626986 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4c8fcd60bc42e496d9ca2067a43115" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:10.627095 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4c8fcd60bc42e496d9ca2067a43115" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:10.627209 28433 raft_consensus.cc:2454] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:10.627302 28043 raft_consensus.cc:2454] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:10.627583 28110 leader_election.cc:302] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:10.630412 29855 raft_consensus.cc:2781] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:10.630507 29855 raft_consensus.cc:482] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:10.630622 29855 raft_consensus.cc:3037] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.632396 29855 raft_consensus.cc:504] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.633028 29855 leader_election.cc:288] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:10.633491 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4c8fcd60bc42e496d9ca2067a43115" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:10.633522 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4c8fcd60bc42e496d9ca2067a43115" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:10.633749 28043 raft_consensus.cc:3037] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.633767 28433 raft_consensus.cc:3037] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.636602 28433 raft_consensus.cc:2454] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:10.636619 28043 raft_consensus.cc:2454] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:10.637017 28110 leader_election.cc:302] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:10.637256 29855 raft_consensus.cc:2781] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:10.637427 29855 raft_consensus.cc:686] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:10.637809 29855 consensus_queue.cc:237] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:10.641034 27897 catalog_manager.cc:5484] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:10.919445 29819 consensus_queue.cc:1035] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:10.924919 29819 consensus_queue.cc:1035] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:10.992153 29903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:10.992449 29903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:11.000381 29903 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 14:13:11.155099 29855 consensus_queue.cc:1035] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:11.217041 29953 consensus_queue.cc:1035] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:11.219399 28041 raft_consensus.cc:1261] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:11.219965 28043 raft_consensus.cc:1261] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:11.223237 29945 consensus_queue.cc:1035] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:11.223829 29945 consensus_queue.cc:1035] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 292.457 ms
time per row: 0.146229 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:12.389313 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:12.398798 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 8868e8d0fd6b4a14970df92960ca0b80 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.398811 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 8868e8d0fd6b4a14970df92960ca0b80 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.399276 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8868e8d0fd6b4a14970df92960ca0b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.399286 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8868e8d0fd6b4a14970df92960ca0b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.402638 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 859456c340dc47bf8fc600ebe8c4b984 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.403015 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859456c340dc47bf8fc600ebe8c4b984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.404480 29972 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:12.404902 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 859456c340dc47bf8fc600ebe8c4b984 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.405059 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 30e4fb0a79814959aa25cf6eb34dfbb9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.405318 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859456c340dc47bf8fc600ebe8c4b984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.405740 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e4fb0a79814959aa25cf6eb34dfbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.406011 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 8868e8d0fd6b4a14970df92960ca0b80 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.406111 29972 tablet_bootstrap.cc:654] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.406948 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 859456c340dc47bf8fc600ebe8c4b984 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.407321 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859456c340dc47bf8fc600ebe8c4b984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.408826 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 30e4fb0a79814959aa25cf6eb34dfbb9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.409303 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e4fb0a79814959aa25cf6eb34dfbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.410897 28152 tablet_service.cc:1467] Processing CreateTablet for tablet 30e4fb0a79814959aa25cf6eb34dfbb9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.411280 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e4fb0a79814959aa25cf6eb34dfbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.411322 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8868e8d0fd6b4a14970df92960ca0b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.423651 29975 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:12.425451 29975 tablet_bootstrap.cc:654] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.427083 29974 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:12.428717 29974 tablet_bootstrap.cc:654] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.433327 29976 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:12.433657 29975 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:12.433965 29975 ts_tablet_manager.cc:1367] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:13:12.435505 29975 raft_consensus.cc:348] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.435878 29976 tablet_bootstrap.cc:654] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.436045 29975 raft_consensus.cc:374] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.436218 29975 raft_consensus.cc:729] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.436513 29975 consensus_queue.cc:260] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.438871 29975 ts_tablet_manager.cc:1398] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:12.439229 29975 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:12.440452 29972 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:12.440632 29975 tablet_bootstrap.cc:654] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.440742 29972 ts_tablet_manager.cc:1367] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.007s
I20240503 14:13:12.441533 29972 raft_consensus.cc:348] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.441880 29972 raft_consensus.cc:374] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.442034 29972 raft_consensus.cc:729] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.442353 29972 consensus_queue.cc:260] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.443871 29974 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:12.444067 29974 ts_tablet_manager.cc:1367] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20240503 14:13:12.444294 29972 ts_tablet_manager.cc:1398] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:12.444669 29972 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:12.444845 29974 raft_consensus.cc:348] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.445338 29974 raft_consensus.cc:374] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.445504 29974 raft_consensus.cc:729] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.445798 29974 consensus_queue.cc:260] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.448069 29972 tablet_bootstrap.cc:654] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.448933 29974 ts_tablet_manager.cc:1398] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:12.449275 29974 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:12.449988 29975 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:12.450181 29975 ts_tablet_manager.cc:1367] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240503 14:13:12.451159 29975 raft_consensus.cc:348] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.451424 29975 raft_consensus.cc:374] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.451500 29975 raft_consensus.cc:729] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.452369 29974 tablet_bootstrap.cc:654] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.452741 29975 consensus_queue.cc:260] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.456035 29972 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:12.456203 29972 ts_tablet_manager.cc:1367] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 14:13:12.456566 29975 ts_tablet_manager.cc:1398] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:12.456868 29972 raft_consensus.cc:348] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.457069 29972 raft_consensus.cc:374] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.457147 29972 raft_consensus.cc:729] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.457425 29972 consensus_queue.cc:260] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.457533 29976 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:12.459203 29972 ts_tablet_manager.cc:1398] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:12.459897 29974 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:12.460114 29974 ts_tablet_manager.cc:1367] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20240503 14:13:12.460600 29976 ts_tablet_manager.cc:1367] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.000s
I20240503 14:13:12.460953 29974 raft_consensus.cc:348] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.461165 29974 raft_consensus.cc:374] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.461248 29974 raft_consensus.cc:729] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.461421 29976 raft_consensus.cc:348] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.461509 29974 consensus_queue.cc:260] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.461627 29976 raft_consensus.cc:374] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.461709 29976 raft_consensus.cc:729] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.461963 29976 consensus_queue.cc:260] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.462776 29974 ts_tablet_manager.cc:1398] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:12.462778 29976 ts_tablet_manager.cc:1398] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:12.463039 29976 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:12.463048 29974 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:12.464884 29976 tablet_bootstrap.cc:654] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.465217 29948 raft_consensus.cc:482] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:12.465358 29948 raft_consensus.cc:504] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.465679 29974 tablet_bootstrap.cc:654] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.466022 29948 leader_election.cc:288] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:12.466789 29976 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:12.466925 29976 ts_tablet_manager.cc:1367] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:12.467053 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e4fb0a79814959aa25cf6eb34dfbb9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:12.467581 29976 raft_consensus.cc:348] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.467772 29976 raft_consensus.cc:374] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.467844 29976 raft_consensus.cc:729] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.468055 29976 consensus_queue.cc:260] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.468231 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e4fb0a79814959aa25cf6eb34dfbb9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
W20240503 14:13:12.468308 28110 leader_election.cc:341] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:12.468436 28433 raft_consensus.cc:2454] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:12.468629 29976 ts_tablet_manager.cc:1398] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:12.468869 28110 leader_election.cc:302] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:13:12.469203 29948 raft_consensus.cc:2781] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:12.469333 29948 raft_consensus.cc:482] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:12.469431 29948 raft_consensus.cc:3037] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.472239 29974 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:12.472384 29974 ts_tablet_manager.cc:1367] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240503 14:13:12.472441 29948 raft_consensus.cc:504] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.473044 29948 leader_election.cc:288] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:12.473070 29974 raft_consensus.cc:348] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.473261 29974 raft_consensus.cc:374] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.473358 29974 raft_consensus.cc:729] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.473462 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e4fb0a79814959aa25cf6eb34dfbb9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:12.473625 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e4fb0a79814959aa25cf6eb34dfbb9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:12.473796 28433 raft_consensus.cc:3037] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.474406 29974 consensus_queue.cc:260] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.474668 28304 raft_consensus.cc:3037] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.474915 29974 ts_tablet_manager.cc:1398] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:12.476203 28433 raft_consensus.cc:2454] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:12.476650 28110 leader_election.cc:302] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:12.477048 29948 raft_consensus.cc:2781] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:12.477118 28304 raft_consensus.cc:2454] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:12.478200 29948 raft_consensus.cc:686] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:12.478510 29948 consensus_queue.cc:237] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:12.482017 27896 catalog_manager.cc:5484] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:12.552920 29939 raft_consensus.cc:482] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:12.553099 29939 raft_consensus.cc:504] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.553833 29939 leader_election.cc:288] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:12.554277 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8868e8d0fd6b4a14970df92960ca0b80" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:12.554384 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8868e8d0fd6b4a14970df92960ca0b80" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:12.554520 28173 raft_consensus.cc:2454] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:12.554621 28042 raft_consensus.cc:2454] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:12.554947 28238 leader_election.cc:302] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:12.555217 29939 raft_consensus.cc:2781] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:12.555311 29939 raft_consensus.cc:482] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:12.555375 29939 raft_consensus.cc:3037] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.557617 29939 raft_consensus.cc:504] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.558228 29939 leader_election.cc:288] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:12.558645 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8868e8d0fd6b4a14970df92960ca0b80" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:12.558682 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8868e8d0fd6b4a14970df92960ca0b80" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:12.558799 28042 raft_consensus.cc:3037] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.558861 28173 raft_consensus.cc:3037] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.560926 28173 raft_consensus.cc:2454] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:12.560931 28042 raft_consensus.cc:2454] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:12.561338 28240 leader_election.cc:302] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:12.561604 29939 raft_consensus.cc:2781] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:12.561758 29939 raft_consensus.cc:686] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:12.562009 29939 consensus_queue.cc:237] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.564944 27896 catalog_manager.cc:5484] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:12.828394 29956 raft_consensus.cc:482] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:12.828567 29956 raft_consensus.cc:504] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.829344 29956 leader_election.cc:288] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:12.829891 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859456c340dc47bf8fc600ebe8c4b984" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:12.829916 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859456c340dc47bf8fc600ebe8c4b984" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:12.830121 28304 raft_consensus.cc:2454] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:12.830123 28433 raft_consensus.cc:2454] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:12.830547 27979 leader_election.cc:302] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:12.830790 29956 raft_consensus.cc:2781] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:12.830911 29956 raft_consensus.cc:482] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:12.831005 29956 raft_consensus.cc:3037] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.832891 29956 raft_consensus.cc:504] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.833376 29956 leader_election.cc:288] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:12.833793 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859456c340dc47bf8fc600ebe8c4b984" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:12.833889 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859456c340dc47bf8fc600ebe8c4b984" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:12.833979 28304 raft_consensus.cc:3037] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.834120 28433 raft_consensus.cc:3037] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.836091 28304 raft_consensus.cc:2454] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:12.836165 28433 raft_consensus.cc:2454] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:12.836417 27979 leader_election.cc:302] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:12.836663 29956 raft_consensus.cc:2781] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:12.837049 29956 raft_consensus.cc:686] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:12.837330 29956 consensus_queue.cc:237] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:12.840142 27896 catalog_manager.cc:5484] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:13.021049 29948 consensus_queue.cc:1035] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.027242 29985 consensus_queue.cc:1035] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.093370 29939 consensus_queue.cc:1035] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.100054 29987 consensus_queue.cc:1035] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:13.206744 29992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:13.207039 29992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:13.214666 29992 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:13.223994 29956 consensus_queue.cc:1035] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.229480 29956 consensus_queue.cc:1035] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
INSERT report
rows total: 2000
time total: 271.502 ms
time per row: 0.135751 ms
I20240503 14:13:13.657353 27902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:13.666558 28152 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4bd9f587564718a017f0ab12d645d5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.666698 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4eb5689908428398f53dc484ea8072 (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.666852 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4bd9f587564718a017f0ab12d645d5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.667052 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4eb5689908428398f53dc484ea8072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.667078 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4bd9f587564718a017f0ab12d645d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.667279 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4bd9f587564718a017f0ab12d645d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.667083 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4eb5689908428398f53dc484ea8072 (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.669039 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4eb5689908428398f53dc484ea8072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.673728 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 6f4bd9f587564718a017f0ab12d645d5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.674145 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f4bd9f587564718a017f0ab12d645d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.675972 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4eb5689908428398f53dc484ea8072 (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.676458 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4eb5689908428398f53dc484ea8072. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.678467 28413 tablet_service.cc:1467] Processing CreateTablet for tablet ee57689f4f764553bbfc9838e19de76d (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.678836 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee57689f4f764553bbfc9838e19de76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.682597 28282 tablet_service.cc:1467] Processing CreateTablet for tablet ee57689f4f764553bbfc9838e19de76d (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.682981 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee57689f4f764553bbfc9838e19de76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.683496 30053 tablet_bootstrap.cc:492] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:13.685770 30053 tablet_bootstrap.cc:654] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.687893 28021 tablet_service.cc:1467] Processing CreateTablet for tablet ee57689f4f764553bbfc9838e19de76d (DEFAULT_TABLE table=rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.688377 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee57689f4f764553bbfc9838e19de76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.690943 30052 tablet_bootstrap.cc:492] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:13.694630 30052 tablet_bootstrap.cc:654] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.696828 30051 tablet_bootstrap.cc:492] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:13.697292 30054 tablet_bootstrap.cc:492] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:13.698758 30051 tablet_bootstrap.cc:654] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.699229 30054 tablet_bootstrap.cc:654] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.699234 30053 tablet_bootstrap.cc:492] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:13.699388 30053 ts_tablet_manager.cc:1367] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 14:13:13.700070 30053 raft_consensus.cc:348] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.700305 30053 raft_consensus.cc:374] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.700407 30053 raft_consensus.cc:729] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.700626 30053 consensus_queue.cc:260] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.701181 30053 ts_tablet_manager.cc:1398] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:13.701309 30051 tablet_bootstrap.cc:492] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:13.701884 30051 ts_tablet_manager.cc:1367] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:13.702170 30053 tablet_bootstrap.cc:492] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:13.702350 30052 tablet_bootstrap.cc:492] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:13.702523 30052 ts_tablet_manager.cc:1367] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 14:13:13.702859 30051 raft_consensus.cc:348] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.703153 30051 raft_consensus.cc:374] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.703294 30051 raft_consensus.cc:729] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.703667 30051 consensus_queue.cc:260] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.703943 30052 raft_consensus.cc:348] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.704293 30052 raft_consensus.cc:374] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.704427 30052 raft_consensus.cc:729] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.704746 30052 consensus_queue.cc:260] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.705514 30052 ts_tablet_manager.cc:1398] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:13.705873 30053 tablet_bootstrap.cc:654] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.705972 30052 tablet_bootstrap.cc:492] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:13.707755 30052 tablet_bootstrap.cc:654] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.709847 30051 ts_tablet_manager.cc:1398] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:13.711465 30051 tablet_bootstrap.cc:492] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:13.714756 30051 tablet_bootstrap.cc:654] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.715137 30054 tablet_bootstrap.cc:492] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:13.715338 30054 ts_tablet_manager.cc:1367] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240503 14:13:13.716369 30054 raft_consensus.cc:348] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.716719 30054 raft_consensus.cc:374] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.716735 30052 tablet_bootstrap.cc:492] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:13.716889 30054 raft_consensus.cc:729] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.716923 30052 ts_tablet_manager.cc:1367] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240503 14:13:13.717285 30054 consensus_queue.cc:260] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.717854 30052 raft_consensus.cc:348] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.718117 30054 ts_tablet_manager.cc:1398] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:13.718199 30052 raft_consensus.cc:374] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.718376 30052 raft_consensus.cc:729] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.718655 30052 consensus_queue.cc:260] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.719012 30054 tablet_bootstrap.cc:492] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:13.719563 30052 ts_tablet_manager.cc:1398] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:13.719928 30052 tablet_bootstrap.cc:492] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:13.720563 30054 tablet_bootstrap.cc:654] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.721462 30053 tablet_bootstrap.cc:492] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:13.721482 30052 tablet_bootstrap.cc:654] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.721675 30053 ts_tablet_manager.cc:1367] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20240503 14:13:13.722601 30053 raft_consensus.cc:348] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.722937 30053 raft_consensus.cc:374] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.723068 30053 raft_consensus.cc:729] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.723441 30053 consensus_queue.cc:260] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.725426 30053 ts_tablet_manager.cc:1398] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:13.731792 30052 tablet_bootstrap.cc:492] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:13.732039 30052 ts_tablet_manager.cc:1367] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.009s
I20240503 14:13:13.733047 30052 raft_consensus.cc:348] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.733378 30052 raft_consensus.cc:374] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.733543 30052 raft_consensus.cc:729] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.733909 30052 consensus_queue.cc:260] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.734700 30052 ts_tablet_manager.cc:1398] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:13.735041 29987 raft_consensus.cc:482] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:13.735208 29987 raft_consensus.cc:504] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.735455 30054 tablet_bootstrap.cc:492] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:13.735600 30054 ts_tablet_manager.cc:1367] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20240503 14:13:13.735780 30051 tablet_bootstrap.cc:492] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:13.735899 30051 ts_tablet_manager.cc:1367] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20240503 14:13:13.736236 29987 leader_election.cc:288] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:13.736462 30054 raft_consensus.cc:348] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.736657 30054 raft_consensus.cc:374] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.736605 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4eb5689908428398f53dc484ea8072" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:13.736764 30054 raft_consensus.cc:729] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.736850 28042 raft_consensus.cc:2454] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:13.737110 30054 consensus_queue.cc:260] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.737274 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4eb5689908428398f53dc484ea8072" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:13.737250 30051 raft_consensus.cc:348] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.737473 30051 raft_consensus.cc:374] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.737469 28173 raft_consensus.cc:2454] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:13.737608 30051 raft_consensus.cc:729] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.737788 30054 ts_tablet_manager.cc:1398] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:13.737886 30051 consensus_queue.cc:260] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.738173 28238 leader_election.cc:302] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:13.738463 30051 ts_tablet_manager.cc:1398] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:13.738540 30035 raft_consensus.cc:2781] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:13.738636 30035 raft_consensus.cc:482] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:13.738736 30035 raft_consensus.cc:3037] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.741425 30035 raft_consensus.cc:504] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.742354 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4eb5689908428398f53dc484ea8072" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:13.742571 28042 raft_consensus.cc:3037] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.743271 30035 leader_election.cc:288] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:13.743727 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4eb5689908428398f53dc484ea8072" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:13.743883 28173 raft_consensus.cc:3037] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.745260 28042 raft_consensus.cc:2454] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:13.746181 28173 raft_consensus.cc:2454] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:13.745651 28240 leader_election.cc:302] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:13.746771 30035 raft_consensus.cc:2781] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:13.746929 30035 raft_consensus.cc:686] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:13.747141 30035 consensus_queue.cc:237] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.749944 27902 catalog_manager.cc:5484] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:13.775107 30035 raft_consensus.cc:482] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:13.775285 30035 raft_consensus.cc:504] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.775905 30035 leader_election.cc:288] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:13.776252 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee57689f4f764553bbfc9838e19de76d" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:13.776357 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee57689f4f764553bbfc9838e19de76d" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:13.776468 28433 raft_consensus.cc:2454] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:13.776592 28042 raft_consensus.cc:2454] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:13.776911 28240 leader_election.cc:302] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:13.777213 29987 raft_consensus.cc:2781] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:13.777325 29987 raft_consensus.cc:482] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:13.777436 29987 raft_consensus.cc:3037] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.779325 29987 raft_consensus.cc:504] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.779798 29987 leader_election.cc:288] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:13.780181 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee57689f4f764553bbfc9838e19de76d" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:13.780251 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee57689f4f764553bbfc9838e19de76d" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:13.780360 28433 raft_consensus.cc:3037] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.780443 28042 raft_consensus.cc:3037] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.782177 28433 raft_consensus.cc:2454] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:13.782289 28042 raft_consensus.cc:2454] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:13.782609 28240 leader_election.cc:302] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:13.783044 29987 raft_consensus.cc:2781] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:13.783216 29987 raft_consensus.cc:686] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:13.783484 29987 consensus_queue.cc:237] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:13.786516 27902 catalog_manager.cc:5484] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:13.816192 30041 raft_consensus.cc:482] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:13.816366 30041 raft_consensus.cc:504] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.817186 30041 leader_election.cc:288] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:13.817513 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4bd9f587564718a017f0ab12d645d5" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:13.817768 28433 raft_consensus.cc:2454] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:13.817730 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4bd9f587564718a017f0ab12d645d5" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:13.817966 28173 raft_consensus.cc:2454] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:13.818157 27979 leader_election.cc:302] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:13.818441 30041 raft_consensus.cc:2781] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:13.818567 30041 raft_consensus.cc:482] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:13.818660 30041 raft_consensus.cc:3037] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.820654 30041 raft_consensus.cc:504] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.821166 30041 leader_election.cc:288] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:13.821487 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4bd9f587564718a017f0ab12d645d5" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:13.821537 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f4bd9f587564718a017f0ab12d645d5" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:13.821677 28433 raft_consensus.cc:3037] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.821736 28173 raft_consensus.cc:3037] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.823573 28173 raft_consensus.cc:2454] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:13.823578 28433 raft_consensus.cc:2454] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:13.823954 27979 leader_election.cc:302] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:13.824240 30041 raft_consensus.cc:2781] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:13.824414 30041 raft_consensus.cc:686] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:13.824672 30041 consensus_queue.cc:237] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:13.827581 27902 catalog_manager.cc:5484] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:14.128333 30035 consensus_queue.cc:1035] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.148358 29987 consensus_queue.cc:1035] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.178985 30035 consensus_queue.cc:1035] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.186755 29987 consensus_queue.cc:1035] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:14.200898 30067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:14.201233 30067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:14.208911 30067 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240503 14:13:14.330857 30041 consensus_queue.cc:1035] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.365048 30041 consensus_queue.cc:1035] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 257.813 ms
time per row: 0.128907 ms
I20240503 14:13:14.630141 27902 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:14.639029 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 213bc70b5cf14440ab88c754df0b8f7f (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.639521 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213bc70b5cf14440ab88c754df0b8f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.639627 28413 tablet_service.cc:1467] Processing CreateTablet for tablet dea30267166942258d47155fd0173713 (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.639966 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea30267166942258d47155fd0173713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.640426 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 213bc70b5cf14440ab88c754df0b8f7f (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.640832 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213bc70b5cf14440ab88c754df0b8f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.640846 28152 tablet_service.cc:1467] Processing CreateTablet for tablet dea30267166942258d47155fd0173713 (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.641273 28152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea30267166942258d47155fd0173713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.643055 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 213bc70b5cf14440ab88c754df0b8f7f (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.643074 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 5f25a3bcdef849499e4a02f01f4dbc0e (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.643414 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213bc70b5cf14440ab88c754df0b8f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.643425 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f25a3bcdef849499e4a02f01f4dbc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.646595 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 5f25a3bcdef849499e4a02f01f4dbc0e (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.646764 28282 tablet_service.cc:1467] Processing CreateTablet for tablet dea30267166942258d47155fd0173713 (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.646906 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f25a3bcdef849499e4a02f01f4dbc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.647104 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea30267166942258d47155fd0173713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.650590 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 5f25a3bcdef849499e4a02f01f4dbc0e (DEFAULT_TABLE table=rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.650940 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f25a3bcdef849499e4a02f01f4dbc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.656306 30139 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:14.658156 30139 tablet_bootstrap.cc:654] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.665511 30139 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:14.666007 30139 ts_tablet_manager.cc:1367] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:13:14.666926 30139 raft_consensus.cc:348] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.668076 30139 raft_consensus.cc:374] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.668321 30139 raft_consensus.cc:729] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.668695 30139 consensus_queue.cc:260] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.669778 30139 ts_tablet_manager.cc:1398] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:14.670487 30139 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:14.672247 30139 tablet_bootstrap.cc:654] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.674165 30138 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:14.677960 30138 tablet_bootstrap.cc:654] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.682463 30141 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:14.682763 30140 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:14.683535 30139 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:14.683684 30139 ts_tablet_manager.cc:1367] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20240503 14:13:14.684278 30140 tablet_bootstrap.cc:654] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.684370 30139 raft_consensus.cc:348] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.684620 30139 raft_consensus.cc:374] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.684701 30139 raft_consensus.cc:729] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.684938 30139 consensus_queue.cc:260] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.685500 30139 ts_tablet_manager.cc:1398] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:14.686137 30141 tablet_bootstrap.cc:654] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.686434 30139 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:14.688527 30138 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:14.688724 30138 ts_tablet_manager.cc:1367] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20240503 14:13:14.689055 30139 tablet_bootstrap.cc:654] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.689472 29948 raft_consensus.cc:482] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:14.689669 29948 raft_consensus.cc:504] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.690367 29948 leader_election.cc:288] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:14.690987 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:14.691485 30139 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:14.691654 30139 ts_tablet_manager.cc:1367] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:14.691605 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
W20240503 14:13:14.691730 28110 leader_election.cc:341] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:13:14.692171 28110 leader_election.cc:341] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:14.692310 28110 leader_election.cc:302] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:13:14.692589 29948 raft_consensus.cc:2726] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:14.692787 30139 raft_consensus.cc:348] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.693073 30139 raft_consensus.cc:374] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.693209 30139 raft_consensus.cc:729] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.693292 30138 raft_consensus.cc:348] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.693535 30138 raft_consensus.cc:374] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.693635 30138 raft_consensus.cc:729] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.693529 30139 consensus_queue.cc:260] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.694319 30139 ts_tablet_manager.cc:1398] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:14.694824 30138 consensus_queue.cc:260] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.698467 30138 ts_tablet_manager.cc:1398] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240503 14:13:14.698802 30138 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:14.699831 30140 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:14.700026 30140 ts_tablet_manager.cc:1367] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240503 14:13:14.700708 30138 tablet_bootstrap.cc:654] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.700642 30140 raft_consensus.cc:348] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.700794 30140 raft_consensus.cc:374] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.700954 30140 raft_consensus.cc:729] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.701172 30140 consensus_queue.cc:260] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.701850 30140 ts_tablet_manager.cc:1398] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:14.702883 30141 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:14.703032 30141 ts_tablet_manager.cc:1367] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240503 14:13:14.703617 30141 raft_consensus.cc:348] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.703765 30141 raft_consensus.cc:374] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.703855 30141 raft_consensus.cc:729] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.704044 30141 consensus_queue.cc:260] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.708173 30141 ts_tablet_manager.cc:1398] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:14.708477 30141 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:14.709134 30138 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:14.709337 30138 ts_tablet_manager.cc:1367] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240503 14:13:14.710060 30138 raft_consensus.cc:348] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.710319 30138 raft_consensus.cc:374] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.710419 30138 raft_consensus.cc:729] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.710673 30138 consensus_queue.cc:260] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.711264 30138 ts_tablet_manager.cc:1398] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:14.711436 30141 tablet_bootstrap.cc:654] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.713215 30141 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:14.713361 30141 ts_tablet_manager.cc:1367] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:14.713970 30141 raft_consensus.cc:348] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.714162 30141 raft_consensus.cc:374] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.714275 30141 raft_consensus.cc:729] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.714490 30141 consensus_queue.cc:260] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.715195 30141 ts_tablet_manager.cc:1398] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:14.715505 30141 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:14.716939 30141 tablet_bootstrap.cc:654] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.718709 30141 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:14.718844 30141 ts_tablet_manager.cc:1367] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:14.719638 30141 raft_consensus.cc:348] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.719777 30141 raft_consensus.cc:374] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.719841 30141 raft_consensus.cc:729] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.719985 30141 consensus_queue.cc:260] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.720381 30141 ts_tablet_manager.cc:1398] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:14.824163 29991 raft_consensus.cc:482] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:14.824430 29991 raft_consensus.cc:504] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.825456 29991 leader_election.cc:288] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:14.825824 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213bc70b5cf14440ab88c754df0b8f7f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:14.826057 28303 raft_consensus.cc:2454] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:14.825987 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213bc70b5cf14440ab88c754df0b8f7f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:14.826212 28174 raft_consensus.cc:2454] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:14.826524 28370 leader_election.cc:302] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:14.826772 29991 raft_consensus.cc:2781] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:14.826866 29991 raft_consensus.cc:482] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:14.826953 29991 raft_consensus.cc:3037] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.828799 29991 raft_consensus.cc:504] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.829413 29991 leader_election.cc:288] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:14.829806 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213bc70b5cf14440ab88c754df0b8f7f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:14.829901 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "213bc70b5cf14440ab88c754df0b8f7f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:14.829994 28303 raft_consensus.cc:3037] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.830098 28174 raft_consensus.cc:3037] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.831831 28303 raft_consensus.cc:2454] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:14.831944 28174 raft_consensus.cc:2454] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:14.832332 28370 leader_election.cc:302] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:14.832664 29991 raft_consensus.cc:2781] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:14.832824 29991 raft_consensus.cc:686] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:14.833061 29991 consensus_queue.cc:237] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.835770 27899 catalog_manager.cc:5484] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:14.849741 30041 raft_consensus.cc:482] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:14.849907 30041 raft_consensus.cc:504] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.850528 30041 leader_election.cc:288] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:14.851006 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25a3bcdef849499e4a02f01f4dbc0e" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:14.851061 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25a3bcdef849499e4a02f01f4dbc0e" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:14.851171 28434 raft_consensus.cc:2454] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:14.851326 28174 raft_consensus.cc:2454] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:14.851702 27979 leader_election.cc:302] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:14.852066 30041 raft_consensus.cc:2781] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:14.852178 30041 raft_consensus.cc:482] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:14.852286 30041 raft_consensus.cc:3037] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.854666 30041 raft_consensus.cc:504] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.855209 30041 leader_election.cc:288] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:14.855633 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25a3bcdef849499e4a02f01f4dbc0e" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:14.855666 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f25a3bcdef849499e4a02f01f4dbc0e" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:14.855824 28174 raft_consensus.cc:3037] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.855826 28434 raft_consensus.cc:3037] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.857981 28434 raft_consensus.cc:2454] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:14.857990 28174 raft_consensus.cc:2454] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:14.858295 27977 leader_election.cc:302] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:14.858665 30041 raft_consensus.cc:2781] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:14.858822 30041 raft_consensus.cc:686] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:14.859064 30041 consensus_queue.cc:237] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:14.861896 27899 catalog_manager.cc:5484] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:14.888777 30035 raft_consensus.cc:482] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:14.889005 30035 raft_consensus.cc:504] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.889659 30035 leader_election.cc:288] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:14.890100 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:14.890125 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:14.890340 28174 raft_consensus.cc:2454] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:14.890353 28434 raft_consensus.cc:2454] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:14.890713 28238 leader_election.cc:302] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:14.890971 30035 raft_consensus.cc:2781] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:14.891072 30035 raft_consensus.cc:482] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:14.891182 30035 raft_consensus.cc:3037] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.893270 30035 raft_consensus.cc:504] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.893873 30035 leader_election.cc:288] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:14.894279 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:14.894315 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:14.894477 28174 raft_consensus.cc:3037] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.894503 28434 raft_consensus.cc:3037] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.897186 28434 raft_consensus.cc:2454] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:14.897212 28174 raft_consensus.cc:2454] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:14.897575 28238 leader_election.cc:302] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:14.897950 30035 raft_consensus.cc:2781] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:14.898118 30035 raft_consensus.cc:686] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:14.898404 30035 consensus_queue.cc:237] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:14.901443 27899 catalog_manager.cc:5484] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:15.244266 29991 consensus_queue.cc:1035] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.250856 30151 consensus_queue.cc:1035] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:15.308485 30155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:15.308827 30155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:13:15.308815 30041 consensus_queue.cc:1035] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.316231 30041 consensus_queue.cc:1035] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:15.318969 30155 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:15.326860 30035 consensus_queue.cc:1035] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.330986 30113 consensus_queue.cc:1035] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 263.518 ms
time per row: 0.131759 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:16.716982 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:16.727993 28021 tablet_service.cc:1467] Processing CreateTablet for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.728461 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.728585 28282 tablet_service.cc:1467] Processing CreateTablet for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.728611 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 8307390c433d4da8bf395b3f6613ca8a (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.728983 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.729049 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8307390c433d4da8bf395b3f6613ca8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.729012 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 8307390c433d4da8bf395b3f6613ca8a (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.729384 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8307390c433d4da8bf395b3f6613ca8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.734908 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 8307390c433d4da8bf395b3f6613ca8a (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.735318 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8307390c433d4da8bf395b3f6613ca8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.736093 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 2055aec2f7284c3c8cb772eca6c1513d (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.736431 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2055aec2f7284c3c8cb772eca6c1513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.741668 28153 tablet_service.cc:1467] Processing CreateTablet for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.742110 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.746727 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 2055aec2f7284c3c8cb772eca6c1513d (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.747181 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2055aec2f7284c3c8cb772eca6c1513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.749147 30224 tablet_bootstrap.cc:492] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:16.753648 30223 tablet_bootstrap.cc:492] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:16.754640 30224 tablet_bootstrap.cc:654] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.755393 30223 tablet_bootstrap.cc:654] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.758787 30228 tablet_bootstrap.cc:492] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:16.763274 30228 tablet_bootstrap.cc:654] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.764448 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 2055aec2f7284c3c8cb772eca6c1513d (DEFAULT_TABLE table=rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:16.765482 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2055aec2f7284c3c8cb772eca6c1513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:16.772679 30228 tablet_bootstrap.cc:492] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:16.773598 30228 ts_tablet_manager.cc:1367] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 14:13:16.775298 30225 tablet_bootstrap.cc:492] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:16.774888 30228 raft_consensus.cc:348] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.777644 30225 tablet_bootstrap.cc:654] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.777665 30228 raft_consensus.cc:374] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.777786 30228 raft_consensus.cc:729] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.778153 30228 consensus_queue.cc:260] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.779142 30228 ts_tablet_manager.cc:1398] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:16.779546 30228 tablet_bootstrap.cc:492] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:16.782045 30228 tablet_bootstrap.cc:654] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.782789 30225 tablet_bootstrap.cc:492] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:16.782934 30225 ts_tablet_manager.cc:1367] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 14:13:16.783690 30225 raft_consensus.cc:348] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.783896 30225 raft_consensus.cc:374] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.783921 30223 tablet_bootstrap.cc:492] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:16.783982 30225 raft_consensus.cc:729] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.784065 30223 ts_tablet_manager.cc:1367] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.032s user 0.015s sys 0.011s
I20240503 14:13:16.784186 30225 consensus_queue.cc:260] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.784737 30225 ts_tablet_manager.cc:1398] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:16.784729 30223 raft_consensus.cc:348] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.784909 30223 raft_consensus.cc:374] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.785066 30223 raft_consensus.cc:729] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.785499 30223 consensus_queue.cc:260] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.785665 30224 tablet_bootstrap.cc:492] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:16.785820 30224 ts_tablet_manager.cc:1367] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.025s
I20240503 14:13:16.786340 30223 ts_tablet_manager.cc:1398] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:16.786648 30223 tablet_bootstrap.cc:492] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:16.786587 30224 raft_consensus.cc:348] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.786790 30224 raft_consensus.cc:374] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.786891 30224 raft_consensus.cc:729] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.787092 30224 consensus_queue.cc:260] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.789119 30224 ts_tablet_manager.cc:1398] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:16.789439 30224 tablet_bootstrap.cc:492] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:16.789479 30223 tablet_bootstrap.cc:654] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.790948 30224 tablet_bootstrap.cc:654] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.794481 30223 tablet_bootstrap.cc:492] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:16.794674 30223 ts_tablet_manager.cc:1367] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240503 14:13:16.795465 30223 raft_consensus.cc:348] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.795737 30223 raft_consensus.cc:374] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.795869 30223 raft_consensus.cc:729] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.796165 30223 consensus_queue.cc:260] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.797134 30228 tablet_bootstrap.cc:492] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:16.797302 30228 ts_tablet_manager.cc:1367] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20240503 14:13:16.797431 30224 tablet_bootstrap.cc:492] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:16.797631 30224 ts_tablet_manager.cc:1367] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240503 14:13:16.797945 30228 raft_consensus.cc:348] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.798148 30228 raft_consensus.cc:374] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.798221 30228 raft_consensus.cc:729] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.798316 30224 raft_consensus.cc:348] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.798559 30224 raft_consensus.cc:374] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.798506 30228 consensus_queue.cc:260] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.798660 30223 ts_tablet_manager.cc:1398] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:13:16.798656 30224 raft_consensus.cc:729] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.798868 30224 consensus_queue.cc:260] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.799018 30228 ts_tablet_manager.cc:1398] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:16.799129 30223 tablet_bootstrap.cc:492] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:16.799474 30224 ts_tablet_manager.cc:1398] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:16.799868 30224 tablet_bootstrap.cc:492] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:16.801167 30223 tablet_bootstrap.cc:654] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.801257 30224 tablet_bootstrap.cc:654] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:16.808113 30223 tablet_bootstrap.cc:492] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:16.808104 30224 tablet_bootstrap.cc:492] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:16.808240 30224 ts_tablet_manager.cc:1367] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240503 14:13:16.808246 30223 ts_tablet_manager.cc:1367] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:13:16.808715 29948 raft_consensus.cc:482] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:16.808914 30224 raft_consensus.cc:348] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.808847 29948 raft_consensus.cc:504] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.809130 30224 raft_consensus.cc:374] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.809038 30223 raft_consensus.cc:348] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.809206 30223 raft_consensus.cc:374] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:16.809211 30224 raft_consensus.cc:729] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.809329 30223 raft_consensus.cc:729] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:16.809430 30224 consensus_queue.cc:260] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.809572 30223 consensus_queue.cc:260] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.809770 29948 leader_election.cc:288] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:16.809973 30224 ts_tablet_manager.cc:1398] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:16.810074 30223 ts_tablet_manager.cc:1398] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:16.810313 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dbe7ee1e7b4dd792a04c87225cb4e8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:16.810504 28303 raft_consensus.cc:2454] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:16.810994 28110 leader_election.cc:302] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:16.811147 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dbe7ee1e7b4dd792a04c87225cb4e8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:16.811350 28043 raft_consensus.cc:2454] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:16.811645 29948 raft_consensus.cc:2781] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:16.811789 29948 raft_consensus.cc:482] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:16.811888 29948 raft_consensus.cc:3037] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.814604 29948 raft_consensus.cc:504] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.815241 29948 leader_election.cc:288] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:16.815651 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dbe7ee1e7b4dd792a04c87225cb4e8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:16.815802 28303 raft_consensus.cc:3037] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.816134 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8dbe7ee1e7b4dd792a04c87225cb4e8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:16.816282 28043 raft_consensus.cc:3037] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.817690 28303 raft_consensus.cc:2454] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:16.818037 28110 leader_election.cc:302] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:16.818379 28043 raft_consensus.cc:2454] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:16.818639 29948 raft_consensus.cc:2781] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:16.818982 29948 raft_consensus.cc:686] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:16.819242 29948 consensus_queue.cc:237] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:16.821761 27899 catalog_manager.cc:5484] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:16.834614 30216 raft_consensus.cc:482] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:16.834749 30216 raft_consensus.cc:504] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.835351 30216 leader_election.cc:288] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:16.835769 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8307390c433d4da8bf395b3f6613ca8a" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:16.835839 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8307390c433d4da8bf395b3f6613ca8a" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:16.836005 28043 raft_consensus.cc:2454] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:16.836079 28303 raft_consensus.cc:2454] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:16.836406 28370 leader_election.cc:302] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:16.836668 30216 raft_consensus.cc:2781] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:16.836792 30216 raft_consensus.cc:482] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:16.836872 30216 raft_consensus.cc:3037] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.838487 29948 raft_consensus.cc:482] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:16.838668 29948 raft_consensus.cc:504] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.838886 30216 raft_consensus.cc:504] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.839244 29948 leader_election.cc:288] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:16.839419 30216 leader_election.cc:288] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:16.839555 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2055aec2f7284c3c8cb772eca6c1513d" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:16.839771 28303 raft_consensus.cc:2454] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:16.840042 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8307390c433d4da8bf395b3f6613ca8a" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:16.840055 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8307390c433d4da8bf395b3f6613ca8a" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:16.840198 28043 raft_consensus.cc:3037] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.840204 28304 raft_consensus.cc:3037] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.842029 28043 raft_consensus.cc:2454] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:16.842048 28304 raft_consensus.cc:2454] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:16.842394 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2055aec2f7284c3c8cb772eca6c1513d" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:16.842419 28370 leader_election.cc:302] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:16.842592 28042 raft_consensus.cc:2454] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:16.842707 30216 raft_consensus.cc:2781] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:16.843034 28110 leader_election.cc:302] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:16.842851 30216 raft_consensus.cc:686] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:16.843307 29948 raft_consensus.cc:2781] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:16.843407 29948 raft_consensus.cc:482] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:16.843551 29948 raft_consensus.cc:3037] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.843549 30216 consensus_queue.cc:237] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:16.845731 29948 raft_consensus.cc:504] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.846323 29948 leader_election.cc:288] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:16.846596 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2055aec2f7284c3c8cb772eca6c1513d" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:16.846697 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2055aec2f7284c3c8cb772eca6c1513d" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:16.846774 28304 raft_consensus.cc:3037] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.846905 28042 raft_consensus.cc:3037] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:16.847162 27899 catalog_manager.cc:5484] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:16.848814 28304 raft_consensus.cc:2454] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:16.849012 28042 raft_consensus.cc:2454] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:16.849418 28110 leader_election.cc:302] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:16.849701 29948 raft_consensus.cc:2781] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:16.849849 29948 raft_consensus.cc:686] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:16.850114 29948 consensus_queue.cc:237] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:16.853085 27899 catalog_manager.cc:5484] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
W20240503 14:13:17.216154 30240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:17.216533 30240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:17.224468 30240 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:17.226739 30236 consensus_queue.cc:1035] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:17.231981 29948 consensus_queue.cc:1035] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
I20240503 14:13:17.271680 30236 consensus_queue.cc:1035] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:17.276479 29948 consensus_queue.cc:1035] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:17.395906 28303 raft_consensus.cc:1261] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:17.396764 30216 consensus_queue.cc:1035] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:17.399632 28043 raft_consensus.cc:1261] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:17.401783 30217 consensus_queue.cc:1035] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 280.643 ms
time per row: 0.140321 ms
I20240503 14:13:17.667268 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:17.676028 28412 tablet_service.cc:1467] Processing CreateTablet for tablet dc9c29880eea4cd9a11eb24eeb3bd228 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.676522 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9c29880eea4cd9a11eb24eeb3bd228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.676427 28154 tablet_service.cc:1467] Processing CreateTablet for tablet dc9c29880eea4cd9a11eb24eeb3bd228 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.676842 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9c29880eea4cd9a11eb24eeb3bd228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.677775 28022 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba7409f5bd4531bc5370019aca14c7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.678171 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba7409f5bd4531bc5370019aca14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.678546 28282 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba7409f5bd4531bc5370019aca14c7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.678959 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba7409f5bd4531bc5370019aca14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.683454 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 664cadf5232f4768adf6438e57f7b7c9 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.683817 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664cadf5232f4768adf6438e57f7b7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.684247 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 664cadf5232f4768adf6438e57f7b7c9 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.684710 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664cadf5232f4768adf6438e57f7b7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.684671 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 664cadf5232f4768adf6438e57f7b7c9 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.685220 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664cadf5232f4768adf6438e57f7b7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.678176 28023 tablet_service.cc:1467] Processing CreateTablet for tablet dc9c29880eea4cd9a11eb24eeb3bd228 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.686560 28413 tablet_service.cc:1467] Processing CreateTablet for tablet d7ba7409f5bd4531bc5370019aca14c7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:17.686672 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9c29880eea4cd9a11eb24eeb3bd228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.686913 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba7409f5bd4531bc5370019aca14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:17.701854 30310 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:17.702486 30311 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:17.704916 30310 tablet_bootstrap.cc:654] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.706749 30311 tablet_bootstrap.cc:654] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.716476 30308 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:17.717108 30309 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:17.718006 30308 tablet_bootstrap.cc:654] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.719832 30308 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:17.720011 30308 ts_tablet_manager.cc:1367] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:17.720414 30310 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:17.720480 30309 tablet_bootstrap.cc:654] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.720580 30310 ts_tablet_manager.cc:1367] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20240503 14:13:17.720772 30308 raft_consensus.cc:348] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.721012 30308 raft_consensus.cc:374] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.721134 30308 raft_consensus.cc:729] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.721189 30311 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:17.721426 30311 ts_tablet_manager.cc:1367] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.003s
I20240503 14:13:17.721606 30308 consensus_queue.cc:260] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.721374 30310 raft_consensus.cc:348] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.722415 30308 ts_tablet_manager.cc:1398] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:17.722447 30310 raft_consensus.cc:374] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.722566 30310 raft_consensus.cc:729] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.722810 30308 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:17.722800 30310 consensus_queue.cc:260] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.723345 30310 ts_tablet_manager.cc:1398] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:17.723703 30311 raft_consensus.cc:348] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.723953 30311 raft_consensus.cc:374] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.724085 30311 raft_consensus.cc:729] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.724483 30310 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:17.724483 30308 tablet_bootstrap.cc:654] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.726300 30310 tablet_bootstrap.cc:654] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.729177 30311 consensus_queue.cc:260] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.734082 30309 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:17.734287 30309 ts_tablet_manager.cc:1367] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20240503 14:13:17.735126 30309 raft_consensus.cc:348] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.735266 30311 ts_tablet_manager.cc:1398] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 14:13:17.735402 30309 raft_consensus.cc:374] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.735550 30309 raft_consensus.cc:729] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.735859 30309 consensus_queue.cc:260] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.736586 30309 ts_tablet_manager.cc:1398] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:17.738425 30310 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:17.738677 30310 ts_tablet_manager.cc:1367] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240503 14:13:17.739671 30309 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:17.739652 30310 raft_consensus.cc:348] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.740006 30310 raft_consensus.cc:374] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.740144 30310 raft_consensus.cc:729] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.740504 30310 consensus_queue.cc:260] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.741206 30309 tablet_bootstrap.cc:654] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.745999 30310 ts_tablet_manager.cc:1398] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:17.746801 30311 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:17.750574 30309 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:17.750672 30311 tablet_bootstrap.cc:654] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.750823 30309 ts_tablet_manager.cc:1367] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 14:13:17.751775 30309 raft_consensus.cc:348] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.752101 30309 raft_consensus.cc:374] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.752190 30309 raft_consensus.cc:729] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.752794 30309 consensus_queue.cc:260] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.754901 30308 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:17.755121 30308 ts_tablet_manager.cc:1367] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.031s
I20240503 14:13:17.755995 30308 raft_consensus.cc:348] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.756263 30308 raft_consensus.cc:374] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.756598 30308 raft_consensus.cc:729] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.756986 30309 ts_tablet_manager.cc:1398] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:17.756922 30308 consensus_queue.cc:260] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.757488 30308 ts_tablet_manager.cc:1398] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:17.757508 30309 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:17.760211 30309 tablet_bootstrap.cc:654] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:17.762122 30309 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:17.762204 30311 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:17.762300 30309 ts_tablet_manager.cc:1367] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:17.762432 30311 ts_tablet_manager.cc:1367] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.001s
I20240503 14:13:17.763013 30311 raft_consensus.cc:348] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.763053 30309 raft_consensus.cc:348] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.763187 30311 raft_consensus.cc:374] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.763221 30309 raft_consensus.cc:374] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:17.763271 30311 raft_consensus.cc:729] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.763303 30309 raft_consensus.cc:729] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:17.763490 30309 consensus_queue.cc:260] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.763520 30311 consensus_queue.cc:260] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.763974 30309 ts_tablet_manager.cc:1398] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:17.764807 30311 ts_tablet_manager.cc:1398] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:17.794153 30041 raft_consensus.cc:482] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:17.794344 30041 raft_consensus.cc:504] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.794950 30041 leader_election.cc:288] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:17.795387 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9c29880eea4cd9a11eb24eeb3bd228" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:17.795409 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9c29880eea4cd9a11eb24eeb3bd228" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:17.795596 28174 raft_consensus.cc:2454] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:17.795603 28434 raft_consensus.cc:2454] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:17.796038 27977 leader_election.cc:302] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:17.796541 30041 raft_consensus.cc:2781] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:17.796672 30041 raft_consensus.cc:482] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:17.796775 30041 raft_consensus.cc:3037] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.798950 30041 raft_consensus.cc:504] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.799448 30041 leader_election.cc:288] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:17.799854 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9c29880eea4cd9a11eb24eeb3bd228" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:17.799916 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc9c29880eea4cd9a11eb24eeb3bd228" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:17.800034 28174 raft_consensus.cc:3037] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.800096 28434 raft_consensus.cc:3037] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.801893 28174 raft_consensus.cc:2454] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:17.801936 28434 raft_consensus.cc:2454] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:17.802230 27977 leader_election.cc:302] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:17.802512 30041 raft_consensus.cc:2781] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:17.802870 30041 raft_consensus.cc:686] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:17.803167 30041 consensus_queue.cc:237] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.806092 27899 catalog_manager.cc:5484] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:17.900230 30303 raft_consensus.cc:482] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:17.900398 30303 raft_consensus.cc:504] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.900992 30303 leader_election.cc:288] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:17.901407 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664cadf5232f4768adf6438e57f7b7c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:17.901407 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664cadf5232f4768adf6438e57f7b7c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:17.901624 28304 raft_consensus.cc:2454] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:17.901624 28174 raft_consensus.cc:2454] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:17.902026 28368 leader_election.cc:302] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:17.902297 30303 raft_consensus.cc:2781] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:17.902401 30303 raft_consensus.cc:482] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:17.902523 30303 raft_consensus.cc:3037] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.904345 30303 raft_consensus.cc:504] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.904857 30303 leader_election.cc:288] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:17.905249 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664cadf5232f4768adf6438e57f7b7c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:17.905297 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "664cadf5232f4768adf6438e57f7b7c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:17.905431 28174 raft_consensus.cc:3037] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.905453 28304 raft_consensus.cc:3037] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.907276 28304 raft_consensus.cc:2454] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:17.907310 28174 raft_consensus.cc:2454] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:17.907675 28370 leader_election.cc:302] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:17.907999 30303 raft_consensus.cc:2781] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:17.908236 30303 raft_consensus.cc:686] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:17.908522 30303 consensus_queue.cc:237] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:17.911795 27899 catalog_manager.cc:5484] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:17.920555 30303 raft_consensus.cc:482] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:17.920728 30303 raft_consensus.cc:504] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.921195 30303 leader_election.cc:288] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:17.921672 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba7409f5bd4531bc5370019aca14c7" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:17.921722 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba7409f5bd4531bc5370019aca14c7" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:17.921912 28304 raft_consensus.cc:2454] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:17.921895 28042 raft_consensus.cc:2454] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:17.922340 28370 leader_election.cc:302] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:17.922590 30303 raft_consensus.cc:2781] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:17.922683 30303 raft_consensus.cc:482] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:17.922962 30303 raft_consensus.cc:3037] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.924922 30303 raft_consensus.cc:504] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.925384 30303 leader_election.cc:288] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:17.925731 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba7409f5bd4531bc5370019aca14c7" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:17.925830 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7ba7409f5bd4531bc5370019aca14c7" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:17.925887 28042 raft_consensus.cc:3037] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.926025 28304 raft_consensus.cc:3037] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:17.927789 28304 raft_consensus.cc:2454] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:17.927829 28042 raft_consensus.cc:2454] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:17.928180 28370 leader_election.cc:302] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:17.928659 30303 raft_consensus.cc:2781] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:17.928823 30303 raft_consensus.cc:686] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:17.929332 30303 consensus_queue.cc:237] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:17.932173 27899 catalog_manager.cc:5484] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:18.320454 30041 consensus_queue.cc:1035] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:18.327750 30041 consensus_queue.cc:1035] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:18.331049 30325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:18.331478 30325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:18.340600 30325 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
I20240503 14:13:18.381757 30217 consensus_queue.cc:1035] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:18.387187 30303 consensus_queue.cc:1035] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:18.440372 30303 consensus_queue.cc:1035] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:18.460018 30303 consensus_queue.cc:1035] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 325.621 ms
time per row: 0.16281 ms
I20240503 14:13:18.816010 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:18.825073 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 05bdc8cd19594b4bab63c8d5ee9131c2 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.825553 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05bdc8cd19594b4bab63c8d5ee9131c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.825474 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 05bdc8cd19594b4bab63c8d5ee9131c2 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.826161 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05bdc8cd19594b4bab63c8d5ee9131c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.826186 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 05bdc8cd19594b4bab63c8d5ee9131c2 (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.826615 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05bdc8cd19594b4bab63c8d5ee9131c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.827049 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee7becef75a4016868696af456e2a3b (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.827392 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee7becef75a4016868696af456e2a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.830926 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee7becef75a4016868696af456e2a3b (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.831323 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee7becef75a4016868696af456e2a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.835676 28284 tablet_service.cc:1467] Processing CreateTablet for tablet ea6b739f48b841e3be03047e84eae57d (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.836050 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6b739f48b841e3be03047e84eae57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.836156 28021 tablet_service.cc:1467] Processing CreateTablet for tablet ea6b739f48b841e3be03047e84eae57d (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.836589 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6b739f48b841e3be03047e84eae57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.840577 30389 tablet_bootstrap.cc:492] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:18.840626 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 3ee7becef75a4016868696af456e2a3b (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.841012 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ee7becef75a4016868696af456e2a3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.844782 28414 tablet_service.cc:1467] Processing CreateTablet for tablet ea6b739f48b841e3be03047e84eae57d (DEFAULT_TABLE table=rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.845311 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea6b739f48b841e3be03047e84eae57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.842664 30389 tablet_bootstrap.cc:654] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.859112 30389 tablet_bootstrap.cc:492] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:18.860091 30389 ts_tablet_manager.cc:1367] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 14:13:18.862032 30389 raft_consensus.cc:348] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:18.863418 30389 raft_consensus.cc:374] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.863771 30389 raft_consensus.cc:729] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.864346 30389 consensus_queue.cc:260] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:18.866190 30389 ts_tablet_manager.cc:1398] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:18.866797 30392 tablet_bootstrap.cc:492] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:18.867563 30389 tablet_bootstrap.cc:492] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:18.868858 30391 tablet_bootstrap.cc:492] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:18.870795 30389 tablet_bootstrap.cc:654] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.871189 30391 tablet_bootstrap.cc:654] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.871397 30392 tablet_bootstrap.cc:654] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.872748 30388 tablet_bootstrap.cc:492] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:18.874329 30388 tablet_bootstrap.cc:654] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.874544 30392 tablet_bootstrap.cc:492] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:18.874682 30392 ts_tablet_manager.cc:1367] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:13:18.875211 30391 tablet_bootstrap.cc:492] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:18.875255 30392 raft_consensus.cc:348] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.875424 30391 ts_tablet_manager.cc:1367] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:13:18.875476 30392 raft_consensus.cc:374] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.875586 30392 raft_consensus.cc:729] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.875808 30392 consensus_queue.cc:260] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.876258 30388 tablet_bootstrap.cc:492] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:18.876387 30392 ts_tablet_manager.cc:1398] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:18.876395 30388 ts_tablet_manager.cc:1367] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:18.876955 30392 tablet_bootstrap.cc:492] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:18.877199 30389 tablet_bootstrap.cc:492] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:18.877391 30389 ts_tablet_manager.cc:1367] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:13:18.877324 30391 raft_consensus.cc:348] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.877460 30388 raft_consensus.cc:348] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.877719 30388 raft_consensus.cc:374] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.877857 30388 raft_consensus.cc:729] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.877610 30391 raft_consensus.cc:374] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.878118 30391 raft_consensus.cc:729] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.878152 30388 consensus_queue.cc:260] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.878171 30389 raft_consensus.cc:348] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.878453 30391 consensus_queue.cc:260] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.878589 30389 raft_consensus.cc:374] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.878746 30389 raft_consensus.cc:729] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.879047 30391 ts_tablet_manager.cc:1398] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:18.879055 30389 consensus_queue.cc:260] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.880023 30392 tablet_bootstrap.cc:654] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.880064 30388 ts_tablet_manager.cc:1398] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:18.880787 30389 ts_tablet_manager.cc:1398] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:18.881196 30388 tablet_bootstrap.cc:492] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:18.885721 30388 tablet_bootstrap.cc:654] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.886888 30391 tablet_bootstrap.cc:492] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:18.888872 30391 tablet_bootstrap.cc:654] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.896420 30388 tablet_bootstrap.cc:492] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:18.896634 30388 ts_tablet_manager.cc:1367] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240503 14:13:18.897603 30388 raft_consensus.cc:348] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.897927 30388 raft_consensus.cc:374] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.898025 30388 raft_consensus.cc:729] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.898355 30388 consensus_queue.cc:260] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.898890 30388 ts_tablet_manager.cc:1398] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:18.899075 30391 tablet_bootstrap.cc:492] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:18.899235 30391 ts_tablet_manager.cc:1367] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240503 14:13:18.899528 30392 tablet_bootstrap.cc:492] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:18.899720 30392 ts_tablet_manager.cc:1367] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.010s
I20240503 14:13:18.899948 30391 raft_consensus.cc:348] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:18.900130 30391 raft_consensus.cc:374] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.900214 30391 raft_consensus.cc:729] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.900451 30391 consensus_queue.cc:260] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:18.900517 30392 raft_consensus.cc:348] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.900781 30392 raft_consensus.cc:374] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.900875 30392 raft_consensus.cc:729] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.900974 30391 ts_tablet_manager.cc:1398] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:18.901156 30392 consensus_queue.cc:260] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.901710 30392 ts_tablet_manager.cc:1398] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:18.902091 30392 tablet_bootstrap.cc:492] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:18.903654 29948 raft_consensus.cc:482] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:18.903800 29948 raft_consensus.cc:504] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.904141 30392 tablet_bootstrap.cc:654] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.904596 29948 leader_election.cc:288] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:18.905416 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bdc8cd19594b4bab63c8d5ee9131c2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:18.905645 28041 raft_consensus.cc:2454] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:18.906371 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bdc8cd19594b4bab63c8d5ee9131c2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:18.906479 28110 leader_election.cc:302] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:18.906582 28434 raft_consensus.cc:2454] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:18.906715 29948 raft_consensus.cc:2781] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:18.906813 29948 raft_consensus.cc:482] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:18.906934 29948 raft_consensus.cc:3037] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.910210 29948 raft_consensus.cc:504] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.910606 30392 tablet_bootstrap.cc:492] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:18.910741 30392 ts_tablet_manager.cc:1367] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240503 14:13:18.910874 29948 leader_election.cc:288] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:18.911159 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bdc8cd19594b4bab63c8d5ee9131c2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:18.911326 28434 raft_consensus.cc:3037] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.911299 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05bdc8cd19594b4bab63c8d5ee9131c2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:18.911440 28041 raft_consensus.cc:3037] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.913391 28041 raft_consensus.cc:2454] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:18.913583 28434 raft_consensus.cc:2454] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:18.913758 28110 leader_election.cc:302] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:18.914032 29948 raft_consensus.cc:2781] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:18.914146 29948 raft_consensus.cc:686] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:18.914294 30392 raft_consensus.cc:348] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:18.914441 30392 raft_consensus.cc:374] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.914438 29948 consensus_queue.cc:237] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.914556 30392 raft_consensus.cc:729] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.914745 30392 consensus_queue.cc:260] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:18.915241 30392 ts_tablet_manager.cc:1398] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:18.917582 27901 catalog_manager.cc:5484] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:18.979348 30113 raft_consensus.cc:482] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:18.979528 30113 raft_consensus.cc:504] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.980144 30113 leader_election.cc:288] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:18.980583 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee7becef75a4016868696af456e2a3b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:18.980592 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee7becef75a4016868696af456e2a3b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:18.980819 28041 raft_consensus.cc:2454] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:18.980854 28174 raft_consensus.cc:2454] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:18.981276 28238 leader_election.cc:302] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:18.981603 30404 raft_consensus.cc:2781] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:18.981766 30404 raft_consensus.cc:482] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:18.981911 30404 raft_consensus.cc:3037] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.984086 30404 raft_consensus.cc:504] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.984612 30404 leader_election.cc:288] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:18.984980 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee7becef75a4016868696af456e2a3b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:18.985172 28173 raft_consensus.cc:3037] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.985131 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ee7becef75a4016868696af456e2a3b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:18.985297 28041 raft_consensus.cc:3037] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.988127 28041 raft_consensus.cc:2454] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:18.988137 28173 raft_consensus.cc:2454] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:18.988512 28240 leader_election.cc:302] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:18.988781 30404 raft_consensus.cc:2781] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:18.989261 30404 raft_consensus.cc:686] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:18.989524 30404 consensus_queue.cc:237] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:18.992832 27901 catalog_manager.cc:5484] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:19.070410 30041 raft_consensus.cc:482] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.070601 30041 raft_consensus.cc:504] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:19.071117 30041 leader_election.cc:288] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:19.071553 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6b739f48b841e3be03047e84eae57d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:19.071578 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6b739f48b841e3be03047e84eae57d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:19.071789 28304 raft_consensus.cc:2454] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:19.071786 28434 raft_consensus.cc:2454] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:19.072166 27979 leader_election.cc:302] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:19.072427 30041 raft_consensus.cc:2781] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.072516 30041 raft_consensus.cc:482] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.072629 30041 raft_consensus.cc:3037] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.074456 30041 raft_consensus.cc:504] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:19.075009 30041 leader_election.cc:288] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:19.075377 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6b739f48b841e3be03047e84eae57d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:19.075485 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea6b739f48b841e3be03047e84eae57d" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:19.075590 28304 raft_consensus.cc:3037] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.075685 28434 raft_consensus.cc:3037] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.077736 28304 raft_consensus.cc:2454] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:19.077911 28434 raft_consensus.cc:2454] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:19.078130 27979 leader_election.cc:302] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:19.078413 30041 raft_consensus.cc:2781] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.078606 30041 raft_consensus.cc:686] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:19.078860 30041 consensus_queue.cc:237] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:19.082015 27901 catalog_manager.cc:5484] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:19.334275 29948 consensus_queue.cc:1035] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.344653 30401 consensus_queue.cc:1035] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.423420 30404 consensus_queue.cc:1035] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.432673 30404 consensus_queue.cc:1035] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:19.490310 30408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:19.490756 30408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:19.498833 30408 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 14:13:19.676074 30041 consensus_queue.cc:1035] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.684230 30379 consensus_queue.cc:1035] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:19.775306 27979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 48826.5us
INSERT report
rows total: 2000
time total: 299.026 ms
time per row: 0.149513 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:20.915467 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:20.924242 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 48266bf4ad9b4976b639553490ffebbe (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.924695 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48266bf4ad9b4976b639553490ffebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.924665 28153 tablet_service.cc:1467] Processing CreateTablet for tablet cd84de9286eb4ad7b88b9f243eb2801c (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.925076 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd84de9286eb4ad7b88b9f243eb2801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.925076 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 48266bf4ad9b4976b639553490ffebbe (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.925474 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48266bf4ad9b4976b639553490ffebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.926187 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 48266bf4ad9b4976b639553490ffebbe (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.926605 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48266bf4ad9b4976b639553490ffebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.927124 28413 tablet_service.cc:1467] Processing CreateTablet for tablet cb88c79b4b1a40c2b45c2dd79ab894d4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.927505 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb88c79b4b1a40c2b45c2dd79ab894d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.928462 28414 tablet_service.cc:1467] Processing CreateTablet for tablet cd84de9286eb4ad7b88b9f243eb2801c (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.928815 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd84de9286eb4ad7b88b9f243eb2801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.933439 28284 tablet_service.cc:1467] Processing CreateTablet for tablet cd84de9286eb4ad7b88b9f243eb2801c (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.933918 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd84de9286eb4ad7b88b9f243eb2801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.937187 28022 tablet_service.cc:1467] Processing CreateTablet for tablet cb88c79b4b1a40c2b45c2dd79ab894d4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.937711 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb88c79b4b1a40c2b45c2dd79ab894d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.943302 28153 tablet_service.cc:1467] Processing CreateTablet for tablet cb88c79b4b1a40c2b45c2dd79ab894d4 (DEFAULT_TABLE table=rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.943694 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb88c79b4b1a40c2b45c2dd79ab894d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.951889 30482 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:20.956346 30482 tablet_bootstrap.cc:654] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.958142 30481 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:20.959998 30481 tablet_bootstrap.cc:654] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.960209 30480 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:20.960747 30483 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:20.961273 30482 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:20.961419 30482 ts_tablet_manager.cc:1367] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:13:20.962224 30482 raft_consensus.cc:348] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.962390 30483 tablet_bootstrap.cc:654] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.962460 30482 raft_consensus.cc:374] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.962577 30482 raft_consensus.cc:729] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.962699 30481 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:20.962846 30481 ts_tablet_manager.cc:1367] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:20.962800 30482 consensus_queue.cc:260] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.963418 30482 ts_tablet_manager.cc:1398] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:20.963833 30482 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:20.964766 30480 tablet_bootstrap.cc:654] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.964795 30481 raft_consensus.cc:348] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.965054 30481 raft_consensus.cc:374] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.965209 30481 raft_consensus.cc:729] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.965287 30482 tablet_bootstrap.cc:654] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.965523 30481 consensus_queue.cc:260] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.966346 30481 ts_tablet_manager.cc:1398] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:13:20.967262 30483 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:20.967393 30483 ts_tablet_manager.cc:1367] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:20.968067 30482 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:20.968211 30482 ts_tablet_manager.cc:1367] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:20.968245 30483 raft_consensus.cc:348] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.968452 30483 raft_consensus.cc:374] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.968544 30483 raft_consensus.cc:729] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.968806 30483 consensus_queue.cc:260] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.968830 30482 raft_consensus.cc:348] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:20.969026 30482 raft_consensus.cc:374] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.969138 30482 raft_consensus.cc:729] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.969414 30482 consensus_queue.cc:260] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:20.969532 30483 ts_tablet_manager.cc:1398] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:20.969892 30482 ts_tablet_manager.cc:1398] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:20.976096 30481 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:20.977335 30481 tablet_bootstrap.cc:654] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.978693 30480 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:20.978816 30480 ts_tablet_manager.cc:1367] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20240503 14:13:20.978839 30481 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:20.978967 30481 ts_tablet_manager.cc:1367] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:20.979494 30481 raft_consensus.cc:348] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:20.979506 30480 raft_consensus.cc:348] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.979640 30480 raft_consensus.cc:374] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.979662 30481 raft_consensus.cc:374] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.979733 30480 raft_consensus.cc:729] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.979732 30481 raft_consensus.cc:729] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.979928 30480 consensus_queue.cc:260] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.979957 30481 consensus_queue.cc:260] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:20.980484 30481 ts_tablet_manager.cc:1398] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:20.980887 30481 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:20.981066 30480 ts_tablet_manager.cc:1398] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:20.981339 30480 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:20.982604 30481 tablet_bootstrap.cc:654] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.983701 30480 tablet_bootstrap.cc:654] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.986974 30481 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:20.987099 30480 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:20.987120 30481 ts_tablet_manager.cc:1367] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240503 14:13:20.987262 30480 ts_tablet_manager.cc:1367] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 14:13:20.987908 30480 raft_consensus.cc:348] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:20.987887 30481 raft_consensus.cc:348] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.988052 30481 raft_consensus.cc:374] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.988073 30480 raft_consensus.cc:374] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.988144 30480 raft_consensus.cc:729] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.988179 30481 raft_consensus.cc:729] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.988384 30480 consensus_queue.cc:260] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:20.988428 30481 consensus_queue.cc:260] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.988890 30481 ts_tablet_manager.cc:1398] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:20.988894 30480 ts_tablet_manager.cc:1398] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:20.989512 30480 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:20.990767 30480 tablet_bootstrap.cc:654] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.992569 30480 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:20.992743 30480 ts_tablet_manager.cc:1367] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:13:20.993353 30480 raft_consensus.cc:348] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.993534 30480 raft_consensus.cc:374] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.993643 30480 raft_consensus.cc:729] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.993861 30480 consensus_queue.cc:260] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:20.994462 30480 ts_tablet_manager.cc:1398] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:21.038115 30475 raft_consensus.cc:482] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.038304 30475 raft_consensus.cc:504] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:21.038945 30475 leader_election.cc:288] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:21.039439 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb88c79b4b1a40c2b45c2dd79ab894d4" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:21.039665 28174 raft_consensus.cc:2454] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:21.039595 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb88c79b4b1a40c2b45c2dd79ab894d4" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:21.039839 28434 raft_consensus.cc:2454] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:21.040247 27979 leader_election.cc:302] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:21.040501 30041 raft_consensus.cc:2781] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:21.040598 30041 raft_consensus.cc:482] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:21.040655 30041 raft_consensus.cc:3037] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.042485 30041 raft_consensus.cc:504] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:21.043015 30041 leader_election.cc:288] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:21.043439 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb88c79b4b1a40c2b45c2dd79ab894d4" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:21.043433 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb88c79b4b1a40c2b45c2dd79ab894d4" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:21.043612 28434 raft_consensus.cc:3037] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.043622 28174 raft_consensus.cc:3037] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.045387 28434 raft_consensus.cc:2454] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:21.045727 27979 leader_election.cc:302] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:21.045958 30041 raft_consensus.cc:2781] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.046095 30041 raft_consensus.cc:686] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:21.046393 30041 consensus_queue.cc:237] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:21.046527 28174 raft_consensus.cc:2454] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:21.049460 27901 catalog_manager.cc:5484] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:21.098249 30303 raft_consensus.cc:482] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.098443 30303 raft_consensus.cc:504] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:21.098989 30303 leader_election.cc:288] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:21.099381 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48266bf4ad9b4976b639553490ffebbe" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:21.099465 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48266bf4ad9b4976b639553490ffebbe" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:21.099642 28173 raft_consensus.cc:2454] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:21.099704 28304 raft_consensus.cc:2454] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:21.100118 28368 leader_election.cc:302] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:21.100405 30303 raft_consensus.cc:2781] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:21.100498 30303 raft_consensus.cc:482] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:21.100608 30303 raft_consensus.cc:3037] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.102572 30303 raft_consensus.cc:504] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:21.103101 30303 leader_election.cc:288] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:21.103482 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48266bf4ad9b4976b639553490ffebbe" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:21.103485 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48266bf4ad9b4976b639553490ffebbe" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:21.103693 28173 raft_consensus.cc:3037] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.103693 28304 raft_consensus.cc:3037] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.106472 28173 raft_consensus.cc:2454] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:21.106491 28304 raft_consensus.cc:2454] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:21.106861 28368 leader_election.cc:302] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:21.107118 30473 raft_consensus.cc:2781] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.107268 30473 raft_consensus.cc:686] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:21.107523 30473 consensus_queue.cc:237] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:21.110816 27901 catalog_manager.cc:5484] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:21.172235 30474 raft_consensus.cc:482] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.172405 30474 raft_consensus.cc:504] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:21.173013 30474 leader_election.cc:288] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:21.173408 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd84de9286eb4ad7b88b9f243eb2801c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:21.173449 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd84de9286eb4ad7b88b9f243eb2801c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:21.173612 28173 raft_consensus.cc:2454] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:21.173645 28434 raft_consensus.cc:2454] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:21.173983 28238 leader_election.cc:302] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:21.174223 30474 raft_consensus.cc:2781] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:21.174364 30474 raft_consensus.cc:482] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:21.174474 30474 raft_consensus.cc:3037] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.176326 30474 raft_consensus.cc:504] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:21.176787 30474 leader_election.cc:288] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:21.177171 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd84de9286eb4ad7b88b9f243eb2801c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:21.177204 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd84de9286eb4ad7b88b9f243eb2801c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:21.177326 28173 raft_consensus.cc:3037] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.177364 28434 raft_consensus.cc:3037] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.179078 28173 raft_consensus.cc:2454] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:21.179198 28434 raft_consensus.cc:2454] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:21.179425 28238 leader_election.cc:302] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:21.179663 30474 raft_consensus.cc:2781] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.179836 30474 raft_consensus.cc:686] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:21.180114 30474 consensus_queue.cc:237] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:21.182981 27901 catalog_manager.cc:5484] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:21.540725 30473 consensus_queue.cc:1035] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.556080 30500 consensus_queue.cc:1035] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:21.571501 30496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:21.571784 30496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:21.579517 30496 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:21.588691 30041 consensus_queue.cc:1035] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.596295 30041 consensus_queue.cc:1035] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
I20240503 14:13:21.782302 28173 raft_consensus.cc:1261] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:21.783171 30452 consensus_queue.cc:1035] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.784894 30474 consensus_queue.cc:1035] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.806 ms
time per row: 0.126903 ms
I20240503 14:13:22.012166 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:22.021342 28022 tablet_service.cc:1467] Processing CreateTablet for tablet fa1b4e565b1244a39a0428646dcb0013 (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.021682 28282 tablet_service.cc:1467] Processing CreateTablet for tablet fa1b4e565b1244a39a0428646dcb0013 (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.021816 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1b4e565b1244a39a0428646dcb0013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.021899 28154 tablet_service.cc:1467] Processing CreateTablet for tablet fa1b4e565b1244a39a0428646dcb0013 (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.022131 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1b4e565b1244a39a0428646dcb0013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.022323 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1b4e565b1244a39a0428646dcb0013. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.022631 28284 tablet_service.cc:1467] Processing CreateTablet for tablet d909ff34d9624201a1085eeb8169200f (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.023005 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d909ff34d9624201a1085eeb8169200f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.026659 28153 tablet_service.cc:1467] Processing CreateTablet for tablet eb0209b00d0e4ee1ac975a60ab500f71 (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.027014 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0209b00d0e4ee1ac975a60ab500f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.031133 28413 tablet_service.cc:1467] Processing CreateTablet for tablet d909ff34d9624201a1085eeb8169200f (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.031775 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d909ff34d9624201a1085eeb8169200f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.031790 30566 tablet_bootstrap.cc:492] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:22.034452 28021 tablet_service.cc:1467] Processing CreateTablet for tablet d909ff34d9624201a1085eeb8169200f (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.034833 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d909ff34d9624201a1085eeb8169200f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.034857 28284 tablet_service.cc:1467] Processing CreateTablet for tablet eb0209b00d0e4ee1ac975a60ab500f71 (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.035187 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0209b00d0e4ee1ac975a60ab500f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.037514 28023 tablet_service.cc:1467] Processing CreateTablet for tablet eb0209b00d0e4ee1ac975a60ab500f71 (DEFAULT_TABLE table=rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.037868 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb0209b00d0e4ee1ac975a60ab500f71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.041730 30566 tablet_bootstrap.cc:654] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.061861 30566 tablet_bootstrap.cc:492] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:22.062022 30566 ts_tablet_manager.cc:1367] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.007s
I20240503 14:13:22.062987 30566 raft_consensus.cc:348] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.063155 30566 raft_consensus.cc:374] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.063253 30566 raft_consensus.cc:729] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.063500 30566 consensus_queue.cc:260] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.064050 30566 ts_tablet_manager.cc:1398] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:22.067553 30566 tablet_bootstrap.cc:492] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:22.073002 30565 tablet_bootstrap.cc:492] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:22.074656 30566 tablet_bootstrap.cc:654] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.077392 30568 tablet_bootstrap.cc:492] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:22.077711 30565 tablet_bootstrap.cc:654] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.078802 30568 tablet_bootstrap.cc:654] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.080050 30565 tablet_bootstrap.cc:492] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:22.080181 30565 ts_tablet_manager.cc:1367] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:22.080845 30565 raft_consensus.cc:348] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.081123 30565 raft_consensus.cc:374] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.081279 30565 raft_consensus.cc:729] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.081466 30569 tablet_bootstrap.cc:492] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:22.081893 30566 tablet_bootstrap.cc:492] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:22.082041 30566 ts_tablet_manager.cc:1367] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240503 14:13:22.082875 30566 raft_consensus.cc:348] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.083092 30566 raft_consensus.cc:374] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.083216 30566 raft_consensus.cc:729] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.083498 30566 consensus_queue.cc:260] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.084084 30566 ts_tablet_manager.cc:1398] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:22.084398 30569 tablet_bootstrap.cc:654] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.084419 30565 consensus_queue.cc:260] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.084933 30566 tablet_bootstrap.cc:492] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:22.085623 30568 tablet_bootstrap.cc:492] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:22.085801 30568 ts_tablet_manager.cc:1367] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:13:22.086587 30566 tablet_bootstrap.cc:654] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.086712 30568 raft_consensus.cc:348] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.086967 30568 raft_consensus.cc:374] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.087100 30568 raft_consensus.cc:729] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.087404 30568 consensus_queue.cc:260] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.087975 30568 ts_tablet_manager.cc:1398] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:22.088299 30568 tablet_bootstrap.cc:492] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:22.088989 30565 ts_tablet_manager.cc:1398] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:13:22.089293 30565 tablet_bootstrap.cc:492] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:22.090807 30565 tablet_bootstrap.cc:654] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.091676 30568 tablet_bootstrap.cc:654] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.096664 30565 tablet_bootstrap.cc:492] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:22.096982 30565 ts_tablet_manager.cc:1367] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 14:13:22.097883 30565 raft_consensus.cc:348] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.100713 30565 raft_consensus.cc:374] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.101122 30565 raft_consensus.cc:729] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.101945 30566 tablet_bootstrap.cc:492] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:22.102123 30566 ts_tablet_manager.cc:1367] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20240503 14:13:22.102420 30565 consensus_queue.cc:260] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.102931 30566 raft_consensus.cc:348] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.103070 30565 ts_tablet_manager.cc:1398] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:13:22.103189 30566 raft_consensus.cc:374] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.103348 30565 tablet_bootstrap.cc:492] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:22.103346 30566 raft_consensus.cc:729] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.103659 30566 consensus_queue.cc:260] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.104897 30565 tablet_bootstrap.cc:654] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.104403 30566 ts_tablet_manager.cc:1398] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:22.109701 30565 tablet_bootstrap.cc:492] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:22.109833 30565 ts_tablet_manager.cc:1367] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240503 14:13:22.110656 30565 raft_consensus.cc:348] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.111243 30565 raft_consensus.cc:374] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.111428 30565 raft_consensus.cc:729] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.111775 30565 consensus_queue.cc:260] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.112584 30565 ts_tablet_manager.cc:1398] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:22.114068 30569 tablet_bootstrap.cc:492] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:22.114276 30569 ts_tablet_manager.cc:1367] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20240503 14:13:22.116246 30569 raft_consensus.cc:348] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.116436 30569 raft_consensus.cc:374] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.116523 30569 raft_consensus.cc:729] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.116732 30569 consensus_queue.cc:260] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.117296 30569 ts_tablet_manager.cc:1398] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:22.118584 30568 tablet_bootstrap.cc:492] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:22.118799 30568 ts_tablet_manager.cc:1367] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.017s
I20240503 14:13:22.119582 30568 raft_consensus.cc:348] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.119979 30568 raft_consensus.cc:374] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.120071 30568 raft_consensus.cc:729] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.120385 30568 consensus_queue.cc:260] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.120883 30568 ts_tablet_manager.cc:1398] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:22.243439 30041 raft_consensus.cc:482] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.243605 30041 raft_consensus.cc:504] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.244141 30041 leader_election.cc:288] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:22.244491 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909ff34d9624201a1085eeb8169200f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:22.244549 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909ff34d9624201a1085eeb8169200f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:22.244683 28304 raft_consensus.cc:2454] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:22.244742 28434 raft_consensus.cc:2454] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:22.245056 27979 leader_election.cc:302] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:22.245301 30041 raft_consensus.cc:2781] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.245442 30041 raft_consensus.cc:482] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.245572 30041 raft_consensus.cc:3037] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.247813 30041 raft_consensus.cc:504] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.248308 30041 leader_election.cc:288] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:22.248644 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909ff34d9624201a1085eeb8169200f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:22.248617 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d909ff34d9624201a1085eeb8169200f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:22.248807 28434 raft_consensus.cc:3037] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.248806 28304 raft_consensus.cc:3037] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.251178 28434 raft_consensus.cc:2454] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:22.251370 28304 raft_consensus.cc:2454] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:22.251608 27979 leader_election.cc:302] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:22.251832 30041 raft_consensus.cc:2781] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.251988 30041 raft_consensus.cc:686] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:22.252219 30041 consensus_queue.cc:237] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:22.255143 27898 catalog_manager.cc:5484] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:22.287400 30468 raft_consensus.cc:482] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.287551 30468 raft_consensus.cc:504] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.288122 30468 leader_election.cc:288] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:22.288482 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b4e565b1244a39a0428646dcb0013" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:22.288532 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b4e565b1244a39a0428646dcb0013" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:22.288671 28041 raft_consensus.cc:2454] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:22.288753 28304 raft_consensus.cc:2454] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:22.289057 28110 leader_election.cc:302] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:22.289281 30468 raft_consensus.cc:2781] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.289387 30468 raft_consensus.cc:482] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.289471 30468 raft_consensus.cc:3037] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.290699 30041 raft_consensus.cc:482] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.290845 30041 raft_consensus.cc:504] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.291590 30041 leader_election.cc:288] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:22.291800 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0209b00d0e4ee1ac975a60ab500f71" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:22.291751 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0209b00d0e4ee1ac975a60ab500f71" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:22.291998 28174 raft_consensus.cc:2454] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:22.292011 28304 raft_consensus.cc:2454] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:22.291780 30468 raft_consensus.cc:504] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.292415 27977 leader_election.cc:302] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:22.292676 30041 raft_consensus.cc:2781] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.292711 30468 leader_election.cc:288] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:22.292790 30041 raft_consensus.cc:482] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.292891 30041 raft_consensus.cc:3037] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.293319 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b4e565b1244a39a0428646dcb0013" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:22.293509 28041 raft_consensus.cc:3037] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.295845 30041 raft_consensus.cc:504] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.296234 28041 raft_consensus.cc:2454] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:22.296604 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1b4e565b1244a39a0428646dcb0013" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:22.296505 30041 leader_election.cc:288] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:22.296936 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0209b00d0e4ee1ac975a60ab500f71" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:22.296932 28110 leader_election.cc:302] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:22.296932 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb0209b00d0e4ee1ac975a60ab500f71" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:22.297075 28302 raft_consensus.cc:3037] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.297199 28174 raft_consensus.cc:3037] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.297298 30468 raft_consensus.cc:2781] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.297474 30468 raft_consensus.cc:686] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:22.297796 30468 consensus_queue.cc:237] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.299633 28174 raft_consensus.cc:2454] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:22.299751 28302 raft_consensus.cc:2454] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:22.300076 28304 raft_consensus.cc:3037] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.300110 27979 leader_election.cc:302] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:22.301144 30041 raft_consensus.cc:2781] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.301633 30041 raft_consensus.cc:686] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:22.301873 27898 catalog_manager.cc:5484] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:22.301918 30041 consensus_queue.cc:237] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:22.302512 28304 raft_consensus.cc:2454] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:22.305390 27898 catalog_manager.cc:5484] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:22.692997 30513 consensus_queue.cc:1035] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.698558 30513 consensus_queue.cc:1035] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:22.701050 30581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:22.701476 30581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:22.710698 30581 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:22.717720 30544 consensus_queue.cc:1035] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.724345 30544 consensus_queue.cc:1035] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.735495 30513 consensus_queue.cc:1035] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.740520 30513 consensus_queue.cc:1035] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
W20240503 14:13:23.074421 28240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.65:33293, user_credentials={real_user=slave}} blocked reactor thread for 53284.5us
INSERT report
rows total: 2000
time total: 342.873 ms
time per row: 0.171437 ms
I20240503 14:13:23.218811 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:23.228286 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 0225203f2de2498eb448e066ce545a64 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.228417 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 0225203f2de2498eb448e066ce545a64 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.228693 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225203f2de2498eb448e066ce545a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.228773 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225203f2de2498eb448e066ce545a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.231058 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 0225203f2de2498eb448e066ce545a64 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.231469 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225203f2de2498eb448e066ce545a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.233331 28154 tablet_service.cc:1467] Processing CreateTablet for tablet edd300244be74e3aa94d7ae35deeb470 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.233697 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd300244be74e3aa94d7ae35deeb470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.233740 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 823dde634aac449ea8cd21d38ecfb22b (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.234127 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823dde634aac449ea8cd21d38ecfb22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.234608 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 823dde634aac449ea8cd21d38ecfb22b (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.234942 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823dde634aac449ea8cd21d38ecfb22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.236222 28414 tablet_service.cc:1467] Processing CreateTablet for tablet edd300244be74e3aa94d7ae35deeb470 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.236590 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd300244be74e3aa94d7ae35deeb470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.238986 28284 tablet_service.cc:1467] Processing CreateTablet for tablet edd300244be74e3aa94d7ae35deeb470 (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.239398 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd300244be74e3aa94d7ae35deeb470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.241351 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 823dde634aac449ea8cd21d38ecfb22b (DEFAULT_TABLE table=rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.241714 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823dde634aac449ea8cd21d38ecfb22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.248675 30652 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:23.252485 30652 tablet_bootstrap.cc:654] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.256565 30651 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:23.258343 30650 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:23.258744 30652 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:23.258827 30651 tablet_bootstrap.cc:654] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.258878 30652 ts_tablet_manager.cc:1367] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:13:23.259737 30652 raft_consensus.cc:348] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.260094 30652 raft_consensus.cc:374] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.260247 30652 raft_consensus.cc:729] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.260551 30650 tablet_bootstrap.cc:654] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.260905 30649 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:23.260987 30652 consensus_queue.cc:260] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.261883 30652 ts_tablet_manager.cc:1398] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:23.262171 30652 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:23.262928 30649 tablet_bootstrap.cc:654] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.263523 30652 tablet_bootstrap.cc:654] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.267664 30651 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:23.267802 30651 ts_tablet_manager.cc:1367] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:13:23.268599 30651 raft_consensus.cc:348] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.268800 30651 raft_consensus.cc:374] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.268913 30651 raft_consensus.cc:729] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.269140 30651 consensus_queue.cc:260] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.269985 30651 ts_tablet_manager.cc:1398] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:23.270591 30651 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:23.271319 30650 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:23.271502 30650 ts_tablet_manager.cc:1367] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240503 14:13:23.272220 30651 tablet_bootstrap.cc:654] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.272586 30650 raft_consensus.cc:348] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.272799 30650 raft_consensus.cc:374] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.272936 30650 raft_consensus.cc:729] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.273236 30650 consensus_queue.cc:260] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.278729 30652 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:23.278956 30652 ts_tablet_manager.cc:1367] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.011s
I20240503 14:13:23.279687 30651 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:23.279891 30651 ts_tablet_manager.cc:1367] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 14:13:23.279917 30652 raft_consensus.cc:348] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.280121 30652 raft_consensus.cc:374] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.280196 30652 raft_consensus.cc:729] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.280408 30652 consensus_queue.cc:260] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.280722 30651 raft_consensus.cc:348] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.280928 30652 ts_tablet_manager.cc:1398] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:23.281810 30651 raft_consensus.cc:374] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.281903 30651 raft_consensus.cc:729] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.282128 30651 consensus_queue.cc:260] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.282721 30651 ts_tablet_manager.cc:1398] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:23.284081 30650 ts_tablet_manager.cc:1398] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 14:13:23.284358 30650 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:23.285936 30650 tablet_bootstrap.cc:654] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.287240 30640 raft_consensus.cc:482] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.287370 30640 raft_consensus.cc:504] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.287896 30649 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:23.288056 30640 leader_election.cc:288] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:23.288069 30649 ts_tablet_manager.cc:1367] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.004s
I20240503 14:13:23.288739 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd300244be74e3aa94d7ae35deeb470" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:23.288969 30649 raft_consensus.cc:348] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.289243 30649 raft_consensus.cc:374] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.289368 30649 raft_consensus.cc:729] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
W20240503 14:13:23.289501 28238 leader_election.cc:341] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.289637 30649 consensus_queue.cc:260] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.290370 30649 ts_tablet_manager.cc:1398] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:23.290762 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd300244be74e3aa94d7ae35deeb470" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:23.291239 30649 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:23.291823 30650 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:23.292042 30650 ts_tablet_manager.cc:1367] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
W20240503 14:13:23.292428 28240 leader_election.cc:341] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.292549 28240 leader_election.cc:302] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f; no voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9
I20240503 14:13:23.292961 30649 tablet_bootstrap.cc:654] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.292927 30650 raft_consensus.cc:348] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.293025 30640 raft_consensus.cc:2726] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:23.293155 30650 raft_consensus.cc:374] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.293226 30650 raft_consensus.cc:729] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.293458 30650 consensus_queue.cc:260] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.294423 30650 ts_tablet_manager.cc:1398] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:23.294744 30650 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:23.297991 30650 tablet_bootstrap.cc:654] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.298044 30649 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:23.298198 30649 ts_tablet_manager.cc:1367] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 14:13:23.299197 30649 raft_consensus.cc:348] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.299427 30649 raft_consensus.cc:374] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.299520 30649 raft_consensus.cc:729] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.299780 30649 consensus_queue.cc:260] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.300114 30650 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:23.300237 30650 ts_tablet_manager.cc:1367] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:23.300326 30649 ts_tablet_manager.cc:1398] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:23.300933 30650 raft_consensus.cc:348] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.301086 30650 raft_consensus.cc:374] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.301188 30650 raft_consensus.cc:729] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.301395 30650 consensus_queue.cc:260] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.301959 30650 ts_tablet_manager.cc:1398] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:23.458405 30500 raft_consensus.cc:482] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.458626 30500 raft_consensus.cc:504] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.459295 30500 leader_election.cc:288] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:23.459637 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823dde634aac449ea8cd21d38ecfb22b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:23.459774 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823dde634aac449ea8cd21d38ecfb22b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:23.459853 28043 raft_consensus.cc:2454] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:23.459985 28304 raft_consensus.cc:2454] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:23.460253 28370 leader_election.cc:302] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:23.460523 30500 raft_consensus.cc:2781] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.460611 30500 raft_consensus.cc:482] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.460726 30500 raft_consensus.cc:3037] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.463725 30500 raft_consensus.cc:504] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.464351 30500 leader_election.cc:288] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:23.464851 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823dde634aac449ea8cd21d38ecfb22b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:23.464959 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "823dde634aac449ea8cd21d38ecfb22b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:23.465051 28304 raft_consensus.cc:3037] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.465135 28043 raft_consensus.cc:3037] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.467686 28043 raft_consensus.cc:2454] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:23.468132 28370 leader_election.cc:302] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:23.468381 30500 raft_consensus.cc:2781] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.468533 28304 raft_consensus.cc:2454] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:23.468586 30500 raft_consensus.cc:686] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:23.468840 30500 consensus_queue.cc:237] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.472215 27896 catalog_manager.cc:5484] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:23.542325 30468 raft_consensus.cc:482] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.542539 30468 raft_consensus.cc:504] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.543224 30468 leader_election.cc:288] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:23.543671 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225203f2de2498eb448e066ce545a64" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:23.543771 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225203f2de2498eb448e066ce545a64" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:23.543882 28432 raft_consensus.cc:2454] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:23.543977 28040 raft_consensus.cc:2454] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:23.544301 28110 leader_election.cc:302] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:23.544575 30468 raft_consensus.cc:2781] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.544690 30468 raft_consensus.cc:482] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.544817 30468 raft_consensus.cc:3037] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.546801 30468 raft_consensus.cc:504] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.547237 30468 leader_election.cc:288] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:23.547605 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225203f2de2498eb448e066ce545a64" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:23.547663 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225203f2de2498eb448e066ce545a64" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:23.547755 28040 raft_consensus.cc:3037] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.547837 28432 raft_consensus.cc:3037] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.549786 28432 raft_consensus.cc:2454] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:23.549849 30500 raft_consensus.cc:482] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.549890 28040 raft_consensus.cc:2454] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:23.549960 30500 raft_consensus.cc:504] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.550297 28110 leader_election.cc:302] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:23.550531 30468 raft_consensus.cc:2781] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.550616 30500 leader_election.cc:288] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:23.551168 30468 raft_consensus.cc:686] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:23.551115 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd300244be74e3aa94d7ae35deeb470" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:23.551257 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd300244be74e3aa94d7ae35deeb470" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:23.551358 28304 raft_consensus.cc:2454] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:23.551482 28172 raft_consensus.cc:2454] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:23.551520 30468 consensus_queue.cc:237] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:23.551857 28370 leader_election.cc:302] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:23.552248 30500 raft_consensus.cc:2781] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.552367 30500 raft_consensus.cc:482] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.552472 30500 raft_consensus.cc:3037] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.555351 27896 catalog_manager.cc:5484] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:23.557010 30500 raft_consensus.cc:504] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.557989 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd300244be74e3aa94d7ae35deeb470" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:23.558224 28304 raft_consensus.cc:3037] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.558373 30500 leader_election.cc:288] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:23.559370 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edd300244be74e3aa94d7ae35deeb470" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:23.559578 28172 raft_consensus.cc:3037] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.560192 28304 raft_consensus.cc:2454] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:23.560545 28370 leader_election.cc:302] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:23.560793 30500 raft_consensus.cc:2781] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.560933 30500 raft_consensus.cc:686] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:23.561205 30500 consensus_queue.cc:237] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:23.561576 28172 raft_consensus.cc:2454] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:23.564497 27896 catalog_manager.cc:5484] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
W20240503 14:13:23.959805 30665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:23.960170 30665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:23.967623 30665 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:23.981420 30303 consensus_queue.cc:1035] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:23.987695 30303 consensus_queue.cc:1035] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_19'
I20240503 14:13:24.018632 30303 consensus_queue.cc:1035] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.024143 30303 consensus_queue.cc:1035] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.035511 30544 consensus_queue.cc:1035] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.054692 30544 consensus_queue.cc:1035] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.35 ms
time per row: 0.126675 ms
I20240503 14:13:24.381734 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:39689
--local_ip_for_outbound_sockets=127.20.240.69
--tserver_master_addrs=127.20.240.126:33033
--webserver_port=38643
--webserver_interface=127.20.240.69
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:13:24.729446 30727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:24.729723 30727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:24.729848 30727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:24.738165 30727 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:24.738389 30727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:24.738549 30727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:13:24.748145 30727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:39689
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=38643
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:24.749464 30727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:24.751127 30727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:24.763126 30733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:24.763604 30735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:24.764300 30732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:24.764952 30727 server_base.cc:1034] running on GCE node
I20240503 14:13:24.765551 30727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:24.766613 30727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:24.767825 30727 hybrid_clock.cc:648] HybridClock initialized: now 1714745604767759 us; error 58 us; skew 500 ppm
I20240503 14:13:24.768106 30727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:24.774999 30727 webserver.cc:460] Webserver started at http://127.20.240.69:38643/ using document root <none> and password file <none>
I20240503 14:13:24.775779 30727 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:24.775923 30727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:24.781144 30727 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 14:13:24.783756 30742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.785336 30727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:24.785513 30727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d0dbfb0e43f948c3922926aacc365fc1"
format_stamp: "Formatted at 2024-05-03 14:12:51 on dist-test-slave-xhmt"
I20240503 14:13:24.786551 30727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:24.806007 30727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:24.806752 30727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:24.806994 30727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:24.807790 30727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:24.809149 30727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:24.809217 30727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.809309 30727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:24.809350 30727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:24.854357 30727 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:39689
I20240503 14:13:24.854411 30854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:39689 every 8 connection(s)
I20240503 14:13:24.855793 30727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:13:24.863373 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 30727
I20240503 14:13:24.864092 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.70:39465
--local_ip_for_outbound_sockets=127.20.240.70
--tserver_master_addrs=127.20.240.126:33033
--webserver_port=39505
--webserver_interface=127.20.240.70
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:24.868749 30855 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:13:24.869028 30855 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:24.869733 30855 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:13:24.871326 27896 ts_manager.cc:194] Re-registered known tserver with Master: d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:24.872336 27896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:39533
W20240503 14:13:25.218084 30859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:25.218369 30859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:25.218484 30859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:25.225857 30859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:25.225987 30859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:25.226138 30859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.70
I20240503 14:13:25.235647 30859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.70:39465
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.240.70
--webserver_port=39505
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.70
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:25.237172 30859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:25.238921 30859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:25.251436 30865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:25.251909 30867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:25.252502 30864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:25.253170 30859 server_base.cc:1034] running on GCE node
I20240503 14:13:25.253718 30859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:25.254642 30859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:25.255834 30859 hybrid_clock.cc:648] HybridClock initialized: now 1714745605255792 us; error 35 us; skew 500 ppm
I20240503 14:13:25.256125 30859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:25.258388 30859 webserver.cc:460] Webserver started at http://127.20.240.70:39505/ using document root <none> and password file <none>
I20240503 14:13:25.259089 30859 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:25.259212 30859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:25.264492 30859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240503 14:13:25.267146 30873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.268412 30859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:25.268563 30859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6bbc686598f24f518d7ca6bb2063058d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:13:25.269297 30859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:25.290153 30859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:25.291011 30859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:25.291309 30859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:25.292343 30859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:25.294148 30859 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:25.294220 30859 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.294363 30859 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:25.294414 30859 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.327865 30859 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.70:39465
I20240503 14:13:25.327919 30985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.70:39465 every 8 connection(s)
I20240503 14:13:25.329365 30859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:13:25.337231 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 30859
I20240503 14:13:25.337904 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.71:34707
--local_ip_for_outbound_sockets=127.20.240.71
--tserver_master_addrs=127.20.240.126:33033
--webserver_port=39749
--webserver_interface=127.20.240.71
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:25.341698 30986 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:13:25.341991 30986 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:25.342738 30986 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:13:25.344311 27896 ts_manager.cc:194] Re-registered known tserver with Master: 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:25.345090 27896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.70:47381
W20240503 14:13:25.688963 30989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:25.689302 30989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:25.689435 30989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:25.697387 30989 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:25.697523 30989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:25.697680 30989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.71
I20240503 14:13:25.706733 30989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.71:34707
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.240.71
--webserver_port=39749
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.71
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:25.708048 30989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:25.709822 30989 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:25.723300 30994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:25.723909 30995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:25.724514 30997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:25.725149 30989 server_base.cc:1034] running on GCE node
I20240503 14:13:25.725718 30989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:25.726680 30989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:25.727874 30989 hybrid_clock.cc:648] HybridClock initialized: now 1714745605727831 us; error 47 us; skew 500 ppm
I20240503 14:13:25.728175 30989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:25.730403 30989 webserver.cc:460] Webserver started at http://127.20.240.71:39749/ using document root <none> and password file <none>
I20240503 14:13:25.731112 30989 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:25.731236 30989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:25.736497 30989 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20240503 14:13:25.739634 31004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.741397 30989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:25.741539 30989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "fde134d0fa2b4900b9badd2e08fea09d"
format_stamp: "Formatted at 2024-05-03 14:12:52 on dist-test-slave-xhmt"
I20240503 14:13:25.742457 30989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:25.800096 30989 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:25.801023 30989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:25.801333 30989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:25.802314 30989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:25.803879 30989 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:25.803977 30989 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.804112 30989 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:25.804165 30989 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:25.852315 30989 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.71:34707
I20240503 14:13:25.852365 31116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.71:34707 every 8 connection(s)
I20240503 14:13:25.853746 30989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:13:25.862175 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 30989
I20240503 14:13:25.862793 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.72:41331
--local_ip_for_outbound_sockets=127.20.240.72
--tserver_master_addrs=127.20.240.126:33033
--webserver_port=40733
--webserver_interface=127.20.240.72
--builtin_ntp_servers=127.20.240.84:36591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:13:25.867900 31117 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:13:25.868235 31117 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:25.869596 31117 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:13:25.871241 27896 ts_manager.cc:194] Re-registered known tserver with Master: fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:25.872515 27896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.71:59809
I20240503 14:13:25.874858 30855 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
W20240503 14:13:26.218556 31121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:26.218824 31121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:26.218953 31121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:26.226606 31121 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:13:26.226738 31121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:26.226886 31121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.72
I20240503 14:13:26.236773 31121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:36591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.72:41331
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.240.72
--webserver_port=40733
--tserver_master_addrs=127.20.240.126:33033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.72
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:26.238049 31121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:26.239866 31121 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:26.251785 31126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:26.252147 31129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:26.252641 31127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:26.253496 31121 server_base.cc:1034] running on GCE node
I20240503 14:13:26.254065 31121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:26.254953 31121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:26.256153 31121 hybrid_clock.cc:648] HybridClock initialized: now 1714745606256100 us; error 50 us; skew 500 ppm
I20240503 14:13:26.256480 31121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:26.258667 31121 webserver.cc:460] Webserver started at http://127.20.240.72:40733/ using document root <none> and password file <none>
I20240503 14:13:26.259346 31121 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:26.259475 31121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:26.264633 31121 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:13:26.267396 31135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:26.268566 31121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:26.268710 31121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "530893a961af4b95b6b687fa5ed22f9d"
format_stamp: "Formatted at 2024-05-03 14:12:53 on dist-test-slave-xhmt"
I20240503 14:13:26.269436 31121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:26.286957 31121 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:26.287726 31121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:26.288028 31121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:26.289098 31121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:26.290686 31121 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:26.290762 31121 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:26.290879 31121 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:26.290935 31121 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:26.344488 31121 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.72:41331
I20240503 14:13:26.344519 31247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.72:41331 every 8 connection(s)
I20240503 14:13:26.346174 31121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:13:26.347579 30986 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
I20240503 14:13:26.349315 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 31121
I20240503 14:13:26.359954 31248 heartbeater.cc:347] Connected to a master server at 127.20.240.126:33033
I20240503 14:13:26.360266 31248 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:26.361117 31248 heartbeater.cc:511] Master 127.20.240.126:33033 requested a full tablet report, sending...
I20240503 14:13:26.362696 27896 ts_manager.cc:194] Re-registered known tserver with Master: 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:26.363942 27896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.72:45413
I20240503 14:13:26.364792 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:26.381003 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1307449a3f4f669811b1964751c3dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903]), partition=
I20240503 14:13:26.381489 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1307449a3f4f669811b1964751c3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.398559 31264 tablet_bootstrap.cc:492] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:26.400569 31264 tablet_bootstrap.cc:654] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.410408 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1307449a3f4f669811b1964751c3dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903]), partition=
I20240503 14:13:26.411422 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1307449a3f4f669811b1964751c3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.418143 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 4f1307449a3f4f669811b1964751c3dd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903]), partition=
I20240503 14:13:26.419348 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f1307449a3f4f669811b1964751c3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.420272 31264 tablet_bootstrap.cc:492] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:26.420480 31264 ts_tablet_manager.cc:1367] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.004s
I20240503 14:13:26.421350 31264 raft_consensus.cc:348] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.421720 31264 raft_consensus.cc:374] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.421880 31264 raft_consensus.cc:729] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.422189 31264 consensus_queue.cc:260] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.422986 31264 ts_tablet_manager.cc:1398] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:26.440205 31266 tablet_bootstrap.cc:492] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:26.442974 31267 tablet_bootstrap.cc:492] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:26.444173 31266 tablet_bootstrap.cc:654] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.445138 31266 log.cc:824] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:26.446223 31267 tablet_bootstrap.cc:654] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.447453 31267 log.cc:824] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:26.450264 31267 tablet_bootstrap.cc:492] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:26.450588 31267 ts_tablet_manager.cc:1367] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240503 14:13:26.451093 31266 tablet_bootstrap.cc:492] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:26.451655 31266 ts_tablet_manager.cc:1367] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:13:26.460608 31266 raft_consensus.cc:348] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.461040 31266 raft_consensus.cc:374] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.461148 31266 raft_consensus.cc:729] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.461907 31266 consensus_queue.cc:260] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.464447 31266 ts_tablet_manager.cc:1398] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.013s user 0.005s sys 0.006s
I20240503 14:13:26.465209 31267 raft_consensus.cc:348] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.465618 31267 raft_consensus.cc:374] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.465767 31267 raft_consensus.cc:729] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.466403 31267 consensus_queue.cc:260] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.468204 31267 ts_tablet_manager.cc:1398] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.017s user 0.008s sys 0.002s
W20240503 14:13:26.581566 30987 tablet.cc:2369] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:13:26.608453 30856 tablet.cc:2369] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:26.611390 31270 raft_consensus.cc:482] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:26.611636 31270 raft_consensus.cc:504] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.622622 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1307449a3f4f669811b1964751c3dd" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:26.623265 30810 raft_consensus.cc:2454] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:26.624040 30874 leader_election.cc:302] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:26.629559 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1307449a3f4f669811b1964751c3dd" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:26.629809 28040 raft_consensus.cc:2454] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:26.632373 31270 leader_election.cc:288] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:26.632582 31270 raft_consensus.cc:2781] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:26.632725 31270 raft_consensus.cc:482] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:26.632824 31270 raft_consensus.cc:3037] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.635778 31270 raft_consensus.cc:504] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.636416 31270 leader_election.cc:288] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:26.637065 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1307449a3f4f669811b1964751c3dd" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:26.637259 28040 raft_consensus.cc:3037] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.638574 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f1307449a3f4f669811b1964751c3dd" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:26.638774 30810 raft_consensus.cc:3037] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.639986 28040 raft_consensus.cc:2454] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:26.640630 30877 leader_election.cc:302] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:26.640982 31270 raft_consensus.cc:2781] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:26.641150 31270 raft_consensus.cc:686] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:26.641721 31270 consensus_queue.cc:237] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:26.644029 30810 raft_consensus.cc:2454] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:26.647105 27896 catalog_manager.cc:5484] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:26.785658 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:26.795209 28040 raft_consensus.cc:1261] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:26.795984 30810 raft_consensus.cc:1261] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:26.796831 31270 consensus_queue.cc:1035] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.797111 31275 consensus_queue.cc:1035] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.804311 27896 catalog_manager.cc:5576] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293) does not have the latest schema for tablet 4f1307449a3f4f669811b1964751c3dd (table rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903]). Expected version 1 got 0
I20240503 14:13:26.804644 31291 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:26.806627 31291 tablet.cc:1620] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:26.816402 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903] alter complete (version 1)
I20240503 14:13:26.820744 31292 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:26.822113 31292 tablet.cc:1620] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:26.829195 27894 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 4f1307449a3f4f669811b1964751c3dd (table rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903]). Expected version 1 got 0
I20240503 14:13:26.829994 27896 catalog_manager.cc:5576] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293) does not have the latest schema for tablet 4f1307449a3f4f669811b1964751c3dd (table rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903]). Expected version 1 got 0
I20240503 14:13:26.830837 31295 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:26.832165 31295 tablet.cc:1620] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:26.851488 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:26.855865 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 14:13:26.856125 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 14:13:26.863418 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 016a1c41112441619700c662beee8c98 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.863879 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet c26fbb09f31c4c8f81788c784da6f205
I20240503 14:13:26.864006 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016a1c41112441619700c662beee8c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.864012 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 016a1c41112441619700c662beee8c98 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.864475 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016a1c41112441619700c662beee8c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.864976 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet c26fbb09f31c4c8f81788c784da6f205 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:26.867569 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 72c7ff14d43342439f935389b03ecad0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.867997 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c7ff14d43342439f935389b03ecad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.873399 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 72c7ff14d43342439f935389b03ecad0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.873955 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c7ff14d43342439f935389b03ecad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.875085 31117 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
I20240503 14:13:26.878487 31266 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:26.880013 31266 tablet_bootstrap.cc:654] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.884893 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 016a1c41112441619700c662beee8c98 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.891261 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet c26fbb09f31c4c8f81788c784da6f205 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:26.891460 28153 tablet_service.cc:1514] Processing DeleteTablet for tablet c26fbb09f31c4c8f81788c784da6f205 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:26.891956 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 5867eb31f7344b178db1a2d0a22da70b
I20240503 14:13:26.893040 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet 5867eb31f7344b178db1a2d0a22da70b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:26.893299 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 5867eb31f7344b178db1a2d0a22da70b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:26.893707 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet aa5031adcf1b4e55a73b615080fbd73c
I20240503 14:13:26.896214 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5031adcf1b4e55a73b615080fbd73c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:26.896333 28022 tablet_service.cc:1514] Processing DeleteTablet for tablet 5867eb31f7344b178db1a2d0a22da70b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:26.896632 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5031adcf1b4e55a73b615080fbd73c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:26.896940 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet aa5031adcf1b4e55a73b615080fbd73c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:26 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:26.900152 28284 tablet_service.cc:1467] Processing CreateTablet for tablet 54a4dbdf6bd64453a9c2c78dbf550448 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.900288 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 54a4dbdf6bd64453a9c2c78dbf550448 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.900761 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a4dbdf6bd64453a9c2c78dbf550448. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.900776 28284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a4dbdf6bd64453a9c2c78dbf550448. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.900792 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016a1c41112441619700c662beee8c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.886384 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 72c7ff14d43342439f935389b03ecad0 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.910828 31267 tablet_bootstrap.cc:492] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:26.912575 31267 tablet_bootstrap.cc:654] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.915513 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c7ff14d43342439f935389b03ecad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.920150 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:26.945858 31291 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:26.946518 31291 tablet.cc:1620] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:26.950407 31295 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:26.950978 31295 tablet.cc:1620] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:26.952564 31292 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:26.953086 31292 tablet.cc:1620] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:26.961146 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=a73a38ee08cc42d9a09fb909a354d903] alter complete (version 2)
I20240503 14:13:26.967017 31266 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:26.967216 31266 ts_tablet_manager.cc:1367] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.089s user 0.000s sys 0.011s
I20240503 14:13:26.968202 31266 raft_consensus.cc:348] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:26.968431 31266 raft_consensus.cc:374] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.968531 31266 raft_consensus.cc:729] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.968734 31266 consensus_queue.cc:260] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:26.969273 31266 ts_tablet_manager.cc:1398] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:26.969544 31266 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:26.975294 31306 tablet_replica.cc:330] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:26.977514 31306 raft_consensus.cc:2229] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:26.977665 31310 tablet_replica.cc:330] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:26.978268 31310 raft_consensus.cc:2229] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:26.981083 31310 raft_consensus.cc:2258] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:26.982419 31310 ts_tablet_manager.cc:1875] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:26.983166 31312 tablet_replica.cc:330] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:26.983727 31312 raft_consensus.cc:2229] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:26.984537 31312 raft_consensus.cc:2258] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:26.984933 31306 raft_consensus.cc:2258] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:26.986027 31267 tablet_bootstrap.cc:492] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:26.986176 31267 ts_tablet_manager.cc:1367] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.012s
I20240503 14:13:26.987978 31267 raft_consensus.cc:348] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:26.988204 31267 raft_consensus.cc:374] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.988298 31267 raft_consensus.cc:729] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.988586 31267 consensus_queue.cc:260] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:26.991539 31306 ts_tablet_manager.cc:1875] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:26.993008 31266 tablet_bootstrap.cc:654] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.993312 31267 ts_tablet_manager.cc:1398] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:26.997867 31267 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:27.002027 31315 tablet_bootstrap.cc:492] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:27.003825 31267 tablet_bootstrap.cc:654] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.007274 31315 tablet_bootstrap.cc:654] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.009639 31311 tablet_replica.cc:330] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:27.009510 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 54a4dbdf6bd64453a9c2c78dbf550448 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.009140 31312 ts_tablet_manager.cc:1875] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.011638 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54a4dbdf6bd64453a9c2c78dbf550448. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.014420 31311 raft_consensus.cc:2229] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.018858 31311 raft_consensus.cc:2258] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.019860 31307 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:27.032001 31320 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:27.032351 31311 ts_tablet_manager.cc:1875] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.039450 31315 tablet_bootstrap.cc:492] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:27.039783 31315 ts_tablet_manager.cc:1367] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.002s
I20240503 14:13:27.040794 31315 raft_consensus.cc:348] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.041146 31315 raft_consensus.cc:374] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.041301 31315 raft_consensus.cc:729] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.041695 31315 consensus_queue.cc:260] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.042783 31315 ts_tablet_manager.cc:1398] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:27.045663 31267 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:27.045862 31267 ts_tablet_manager.cc:1367] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.048s user 0.011s sys 0.013s
I20240503 14:13:27.046808 31267 raft_consensus.cc:348] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.047192 31267 raft_consensus.cc:374] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.047291 31267 raft_consensus.cc:729] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.047649 31267 consensus_queue.cc:260] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.048326 31266 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:27.048547 31266 ts_tablet_manager.cc:1367] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.001s
I20240503 14:13:27.050662 31266 raft_consensus.cc:348] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.050966 31266 raft_consensus.cc:374] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.051128 31266 raft_consensus.cc:729] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.051493 31266 consensus_queue.cc:260] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.052783 31267 ts_tablet_manager.cc:1398] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:27.055014 31307 tablet_bootstrap.cc:654] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.060820 31306 ts_tablet_manager.cc:1888] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.061105 31306 log.cc:1196] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/c26fbb09f31c4c8f81788c784da6f205
I20240503 14:13:27.061996 31306 ts_tablet_manager.cc:1909] T c26fbb09f31c4c8f81788c784da6f205 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:27.065610 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:27.066427 31310 ts_tablet_manager.cc:1888] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.066661 31310 log.cc:1196] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/c26fbb09f31c4c8f81788c784da6f205
I20240503 14:13:27.067238 31312 ts_tablet_manager.cc:1888] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.070840 31310 ts_tablet_manager.cc:1909] T c26fbb09f31c4c8f81788c784da6f205 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.072679 31312 log.cc:1196] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/5867eb31f7344b178db1a2d0a22da70b
I20240503 14:13:27.078557 31312 ts_tablet_manager.cc:1909] T 5867eb31f7344b178db1a2d0a22da70b P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:27.080502 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet c26fbb09f31c4c8f81788c784da6f205 (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.082372 31311 ts_tablet_manager.cc:1888] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.083643 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 5867eb31f7344b178db1a2d0a22da70b (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.084034 31310 tablet_replica.cc:330] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.084301 31310 raft_consensus.cc:2229] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.084560 31310 raft_consensus.cc:2258] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.084841 31312 tablet_replica.cc:330] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:27.085773 31312 raft_consensus.cc:2229] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.086079 31312 raft_consensus.cc:2258] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.086293 31311 log.cc:1196] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/c26fbb09f31c4c8f81788c784da6f205
I20240503 14:13:27.087177 31311 ts_tablet_manager.cc:1909] T c26fbb09f31c4c8f81788c784da6f205 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:27.087342 31310 ts_tablet_manager.cc:1875] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.088768 31312 ts_tablet_manager.cc:1875] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.089346 31292 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:27.093714 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet c26fbb09f31c4c8f81788c784da6f205 (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.095293 31311 tablet_replica.cc:330] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:27.095431 31295 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:27.095700 31311 raft_consensus.cc:2229] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.095897 31311 raft_consensus.cc:2258] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.100116 31311 ts_tablet_manager.cc:1875] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.101032 31312 ts_tablet_manager.cc:1888] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.101251 31312 log.cc:1196] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa5031adcf1b4e55a73b615080fbd73c
I20240503 14:13:27.101713 31312 ts_tablet_manager.cc:1909] T aa5031adcf1b4e55a73b615080fbd73c P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:27.104393 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet aa5031adcf1b4e55a73b615080fbd73c (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.085937 31291 tablet.cc:1722] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:27.107263 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=a73a38ee08cc42d9a09fb909a354d903] alter complete (version 3)
I20240503 14:13:27.108377 31311 ts_tablet_manager.cc:1888] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.108516 31311 log.cc:1196] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/5867eb31f7344b178db1a2d0a22da70b
I20240503 14:13:27.109074 31311 ts_tablet_manager.cc:1909] T 5867eb31f7344b178db1a2d0a22da70b P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:27.110956 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 5867eb31f7344b178db1a2d0a22da70b (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.111884 31266 ts_tablet_manager.cc:1398] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20240503 14:13:27.112365 31310 ts_tablet_manager.cc:1888] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.112510 31310 log.cc:1196] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/5867eb31f7344b178db1a2d0a22da70b
I20240503 14:13:27.112931 31310 ts_tablet_manager.cc:1909] T 5867eb31f7344b178db1a2d0a22da70b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.114197 31320 tablet_bootstrap.cc:654] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.114295 31310 tablet_replica.cc:330] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.114665 31310 raft_consensus.cc:2229] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.114976 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 5867eb31f7344b178db1a2d0a22da70b (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.115037 31310 raft_consensus.cc:2258] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.115933 31310 ts_tablet_manager.cc:1875] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.117332 31320 log.cc:824] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:27.117671 31306 tablet_replica.cc:330] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:27.117693 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet c26fbb09f31c4c8f81788c784da6f205 (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.117930 31306 raft_consensus.cc:2229] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.118175 31306 raft_consensus.cc:2258] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.118886 31306 ts_tablet_manager.cc:1875] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.122172 31320 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:27.113247 31307 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:27.122375 31307 ts_tablet_manager.cc:1367] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.103s user 0.000s sys 0.004s
I20240503 14:13:27.123271 31307 raft_consensus.cc:348] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:27.123485 31307 raft_consensus.cc:374] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.123594 31307 raft_consensus.cc:729] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.123914 31320 ts_tablet_manager.cc:1367] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.000s
I20240503 14:13:27.123822 31307 consensus_queue.cc:260] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:27.131749 31307 ts_tablet_manager.cc:1398] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:13:27.132994 31320 raft_consensus.cc:348] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:27.133473 31320 raft_consensus.cc:374] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.133703 31320 raft_consensus.cc:729] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.134546 31320 consensus_queue.cc:260] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:27.136842 31275 raft_consensus.cc:482] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.137027 31275 raft_consensus.cc:504] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.137889 31275 leader_election.cc:288] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:27.138211 31248 heartbeater.cc:503] Master 127.20.240.126:33033 was elected leader, sending a full tablet report...
I20240503 14:13:27.138428 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c7ff14d43342439f935389b03ecad0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:27.138638 30809 raft_consensus.cc:2454] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:27.139041 30874 leader_election.cc:302] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:27.139382 31275 raft_consensus.cc:2781] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.139461 31275 raft_consensus.cc:482] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.139541 31275 raft_consensus.cc:3037] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.140376 31306 ts_tablet_manager.cc:1888] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.140517 31306 log.cc:1196] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/aa5031adcf1b4e55a73b615080fbd73c
I20240503 14:13:27.140936 31306 ts_tablet_manager.cc:1909] T aa5031adcf1b4e55a73b615080fbd73c P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:27.130192 31310 ts_tablet_manager.cc:1888] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.141840 31310 log.cc:1196] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/aa5031adcf1b4e55a73b615080fbd73c
I20240503 14:13:27.142093 31275 raft_consensus.cc:504] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.142549 31310 ts_tablet_manager.cc:1909] T aa5031adcf1b4e55a73b615080fbd73c P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.142838 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet aa5031adcf1b4e55a73b615080fbd73c (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.143240 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c7ff14d43342439f935389b03ecad0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:27.143424 30809 raft_consensus.cc:3037] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.143864 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet aa5031adcf1b4e55a73b615080fbd73c (table rebalance_test_table_0 [id=7fb825b0755544ab9ec8dc7f918ce8dd]) successfully deleted
I20240503 14:13:27.145114 31337 tablet_bootstrap.cc:492] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:27.146409 30809 raft_consensus.cc:2454] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:27.146857 30874 leader_election.cc:302] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:27.147154 31270 raft_consensus.cc:2781] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.147141 30474 raft_consensus.cc:482] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.147323 30474 raft_consensus.cc:504] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.148361 30474 leader_election.cc:288] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:27.148845 31275 leader_election.cc:288] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:27.151137 31337 tablet_bootstrap.cc:654] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.152731 31337 log.cc:824] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:27.156217 31320 ts_tablet_manager.cc:1398] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.032s user 0.014s sys 0.000s
I20240503 14:13:27.157559 31320 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:27.160919 31270 raft_consensus.cc:686] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:27.161406 31270 consensus_queue.cc:237] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.162791 31320 tablet_bootstrap.cc:654] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.168226 31320 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:27.168920 31320 ts_tablet_manager.cc:1367] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:13:27.170766 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c7ff14d43342439f935389b03ecad0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:27.171265 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c7ff14d43342439f935389b03ecad0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:27.170020 31320 raft_consensus.cc:348] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.173918 31320 raft_consensus.cc:374] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.174034 31320 raft_consensus.cc:729] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.174322 31320 consensus_queue.cc:260] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:27.174723 31203 raft_consensus.cc:2454] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:27.174665 27896 catalog_manager.cc:5484] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:27.174921 31320 ts_tablet_manager.cc:1398] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:27.182054 31337 tablet_bootstrap.cc:492] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:27.182385 31337 ts_tablet_manager.cc:1367] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.004s
I20240503 14:13:27.191741 31337 raft_consensus.cc:348] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.192260 31337 raft_consensus.cc:374] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.192363 31337 raft_consensus.cc:729] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.193056 31337 consensus_queue.cc:260] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.194918 31337 ts_tablet_manager.cc:1398] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240503 14:13:27.202065 31346 raft_consensus.cc:482] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.202442 31346 raft_consensus.cc:504] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.207247 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:27.207501 30809 raft_consensus.cc:2454] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:27.210036 31346 leader_election.cc:288] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:27.212054 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:27.212261 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:27.216800 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 4f1307449a3f4f669811b1964751c3dd
I20240503 14:13:27.217001 28237 leader_election.cc:302] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:27.217542 30682 raft_consensus.cc:2781] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.217640 30682 raft_consensus.cc:482] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.217739 30682 raft_consensus.cc:3037] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.217698 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:27.217927 30809 raft_consensus.cc:2454] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:27.218562 31005 leader_election.cc:302] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:27.218698 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1307449a3f4f669811b1964751c3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:27.220511 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1307449a3f4f669811b1964751c3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:27.220853 30682 raft_consensus.cc:504] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.221520 30682 leader_election.cc:288] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:27.224925 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f1307449a3f4f669811b1964751c3dd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:27.225176 31310 tablet_replica.cc:330] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.225395 31310 raft_consensus.cc:2229] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.225595 31310 raft_consensus.cc:2258] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.226773 31310 ts_tablet_manager.cc:1875] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.230583 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:27.231004 31072 raft_consensus.cc:2454] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:27.231460 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:27.231725 31071 raft_consensus.cc:3037] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.235335 31071 raft_consensus.cc:2454] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:27.235790 28239 leader_election.cc:302] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:27.236459 31346 raft_consensus.cc:2741] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 14:13:27.236649 31310 ts_tablet_manager.cc:1888] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:27.236761 30682 raft_consensus.cc:2781] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.236958 31310 log.cc:1196] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/4f1307449a3f4f669811b1964751c3dd
I20240503 14:13:27.237288 30682 raft_consensus.cc:686] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:27.237627 30682 consensus_queue.cc:237] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:27.239533 31310 ts_tablet_manager.cc:1909] T 4f1307449a3f4f669811b1964751c3dd P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.240762 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 4f1307449a3f4f669811b1964751c3dd (table rebalancer_dynamic_table_new_name [id=a73a38ee08cc42d9a09fb909a354d903]) successfully deleted
I20240503 14:13:27.241775 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:27.246405 31270 raft_consensus.cc:482] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.246605 31270 raft_consensus.cc:504] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:27.247589 31270 leader_election.cc:288] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:27.248405 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016a1c41112441619700c662beee8c98" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:27.248639 31203 raft_consensus.cc:2454] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:27.249702 30875 leader_election.cc:302] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:27.250031 31270 raft_consensus.cc:2781] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.250131 31270 raft_consensus.cc:482] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.250257 31270 raft_consensus.cc:3037] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.253849 31270 raft_consensus.cc:504] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:27.255511 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016a1c41112441619700c662beee8c98" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:27.255736 31203 raft_consensus.cc:3037] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.260054 31203 raft_consensus.cc:2454] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:27.260613 30875 leader_election.cc:302] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:27.260991 31339 raft_consensus.cc:2781] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.261211 31339 raft_consensus.cc:686] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:27.261643 31339 consensus_queue.cc:237] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:27.261494 31270 leader_election.cc:288] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:27.265550 27899 catalog_manager.cc:5484] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:27.270714 27896 catalog_manager.cc:5484] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:27.281029 31351 tablet_replica.cc:330] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:27.281613 31351 raft_consensus.cc:2229] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.282363 31351 raft_consensus.cc:2258] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.282407 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016a1c41112441619700c662beee8c98" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:27.282658 28432 raft_consensus.cc:3037] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.283035 31352 tablet_replica.cc:330] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:27.283478 31352 raft_consensus.cc:2229] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.283805 31352 raft_consensus.cc:2258] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:27.285343 28237 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 68401.6us
I20240503 14:13:27.285447 28432 raft_consensus.cc:2454] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:27.285910 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016a1c41112441619700c662beee8c98" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:27.285938 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:27.286124 28434 raft_consensus.cc:2362] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1. Re-sending same reply.
I20240503 14:13:27.286157 31351 ts_tablet_manager.cc:1875] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.286152 30809 raft_consensus.cc:3037] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.287637 31352 ts_tablet_manager.cc:1875] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.288836 30809 raft_consensus.cc:2454] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:27.291800 31351 ts_tablet_manager.cc:1888] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:27.291915 31351 log.cc:1196] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/4f1307449a3f4f669811b1964751c3dd
I20240503 14:13:27.292338 31351 ts_tablet_manager.cc:1909] T 4f1307449a3f4f669811b1964751c3dd P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:27.292834 31352 ts_tablet_manager.cc:1888] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:27.292958 31352 log.cc:1196] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/4f1307449a3f4f669811b1964751c3dd
I20240503 14:13:27.293394 31352 ts_tablet_manager.cc:1909] T 4f1307449a3f4f669811b1964751c3dd P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:27.293690 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 4f1307449a3f4f669811b1964751c3dd (table rebalancer_dynamic_table_new_name [id=a73a38ee08cc42d9a09fb909a354d903]) successfully deleted
I20240503 14:13:27.295007 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 4f1307449a3f4f669811b1964751c3dd (table rebalancer_dynamic_table_new_name [id=a73a38ee08cc42d9a09fb909a354d903]) successfully deleted
I20240503 14:13:27.321367 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:27.330634 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 22ae0dc65b694823939ced26668ae1c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45]), partition=
I20240503 14:13:27.330840 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 22ae0dc65b694823939ced26668ae1c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45]), partition=
I20240503 14:13:27.331055 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22ae0dc65b694823939ced26668ae1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.331223 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22ae0dc65b694823939ced26668ae1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.336438 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 22ae0dc65b694823939ced26668ae1c6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45]), partition=
I20240503 14:13:27.336838 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22ae0dc65b694823939ced26668ae1c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.342108 31266 tablet_bootstrap.cc:492] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:27.343756 31266 tablet_bootstrap.cc:654] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.345018 31267 tablet_bootstrap.cc:492] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:27.346647 31267 tablet_bootstrap.cc:654] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:27.348893 31249 tablet.cc:2369] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:27.355749 31320 tablet_bootstrap.cc:492] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:27.357513 31320 tablet_bootstrap.cc:654] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:27.358541 31118 tablet.cc:2369] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:27.368430 31320 tablet_bootstrap.cc:492] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:27.368618 31320 ts_tablet_manager.cc:1367] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:13:27.369485 31320 raft_consensus.cc:348] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.369822 31320 raft_consensus.cc:374] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.369910 31320 raft_consensus.cc:729] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.370203 31266 tablet_bootstrap.cc:492] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:27.370395 31266 ts_tablet_manager.cc:1367] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.004s
I20240503 14:13:27.370247 31320 consensus_queue.cc:260] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.371587 31320 ts_tablet_manager.cc:1398] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:27.371668 31266 raft_consensus.cc:348] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.371868 31266 raft_consensus.cc:374] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.371958 31266 raft_consensus.cc:729] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.372243 31266 consensus_queue.cc:260] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.372828 31266 ts_tablet_manager.cc:1398] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:27.376399 31267 tablet_bootstrap.cc:492] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:27.376731 31267 ts_tablet_manager.cc:1367] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.006s
I20240503 14:13:27.377635 31267 raft_consensus.cc:348] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.377908 31267 raft_consensus.cc:374] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.378018 31267 raft_consensus.cc:729] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.378290 31267 consensus_queue.cc:260] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.379858 31267 ts_tablet_manager.cc:1398] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:27.405501 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 14:13:27.405710 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 14:13:27.409722 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 4d05336747e94520bccc53cdb85e51b7
I20240503 14:13:27.410303 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet ef0b6bb524e842f5ade540e691e50a97
I20240503 14:13:27.410714 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 07f07ce659e844e3afc090a0753c1128
I20240503 14:13:27.411662 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0b6bb524e842f5ade540e691e50a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:27.411847 28413 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d05336747e94520bccc53cdb85e51b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:27.411926 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0b6bb524e842f5ade540e691e50a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:27.412058 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f07ce659e844e3afc090a0753c1128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:27.412063 28022 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f07ce659e844e3afc090a0753c1128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:27.412356 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d05336747e94520bccc53cdb85e51b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:27.412411 28153 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d05336747e94520bccc53cdb85e51b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:27.412569 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet ef0b6bb524e842f5ade540e691e50a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:27.412550 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 07f07ce659e844e3afc090a0753c1128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:27.412595 31312 tablet_replica.cc:330] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:27.412748 31312 raft_consensus.cc:2229] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.412822 31306 tablet_replica.cc:330] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:27.412957 31312 raft_consensus.cc:2258] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.413095 31306 raft_consensus.cc:2229] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.413496 31312 ts_tablet_manager.cc:1875] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.413710 31306 raft_consensus.cc:2258] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.414664 31306 ts_tablet_manager.cc:1875] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.416664 31310 tablet_replica.cc:330] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.416944 31310 raft_consensus.cc:2229] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.417279 31310 raft_consensus.cc:2258] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.418145 31310 ts_tablet_manager.cc:1875] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.418390 31311 tablet_replica.cc:330] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:27.418576 31311 raft_consensus.cc:2229] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.418831 31311 raft_consensus.cc:2258] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.419353 31311 ts_tablet_manager.cc:1875] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.425341 31311 ts_tablet_manager.cc:1888] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.425467 31311 log.cc:1196] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/4d05336747e94520bccc53cdb85e51b7
I20240503 14:13:27.425868 31311 ts_tablet_manager.cc:1909] T 4d05336747e94520bccc53cdb85e51b7 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:27.426390 31312 ts_tablet_manager.cc:1888] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.426565 31312 log.cc:1196] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/ef0b6bb524e842f5ade540e691e50a97
I20240503 14:13:27.426826 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 4d05336747e94520bccc53cdb85e51b7 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.427162 31311 tablet_replica.cc:330] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:27.427166 31312 ts_tablet_manager.cc:1909] T ef0b6bb524e842f5ade540e691e50a97 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:27.427448 31311 raft_consensus.cc:2229] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.427971 31311 raft_consensus.cc:2258] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.428206 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet ef0b6bb524e842f5ade540e691e50a97 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.432129 31311 ts_tablet_manager.cc:1875] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.432435 31310 ts_tablet_manager.cc:1888] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.432595 31310 log.cc:1196] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/07f07ce659e844e3afc090a0753c1128
I20240503 14:13:27.433156 31310 ts_tablet_manager.cc:1909] T 07f07ce659e844e3afc090a0753c1128 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.434293 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 07f07ce659e844e3afc090a0753c1128 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.436317 31311 ts_tablet_manager.cc:1888] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.436447 31311 log.cc:1196] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/07f07ce659e844e3afc090a0753c1128
I20240503 14:13:27.436869 31311 ts_tablet_manager.cc:1909] T 07f07ce659e844e3afc090a0753c1128 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:27.437646 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 07f07ce659e844e3afc090a0753c1128 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.438177 31306 ts_tablet_manager.cc:1888] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.438449 31306 log.cc:1196] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/ef0b6bb524e842f5ade540e691e50a97
I20240503 14:13:27.438369 31310 tablet_replica.cc:330] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.439224 31306 ts_tablet_manager.cc:1909] T ef0b6bb524e842f5ade540e691e50a97 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:27.439424 31310 raft_consensus.cc:2229] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.439765 31310 raft_consensus.cc:2258] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.440560 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet ef0b6bb524e842f5ade540e691e50a97 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.440616 31310 ts_tablet_manager.cc:1875] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.440659 31306 tablet_replica.cc:330] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:27.440878 31306 raft_consensus.cc:2229] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.441210 31306 raft_consensus.cc:2258] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.442196 31306 ts_tablet_manager.cc:1875] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.446794 31310 ts_tablet_manager.cc:1888] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.447021 31310 log.cc:1196] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/4d05336747e94520bccc53cdb85e51b7
I20240503 14:13:27.447499 31310 ts_tablet_manager.cc:1909] T 4d05336747e94520bccc53cdb85e51b7 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.448704 31310 tablet_replica.cc:330] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.449115 31310 raft_consensus.cc:2229] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.449168 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 4d05336747e94520bccc53cdb85e51b7 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.449512 31310 raft_consensus.cc:2258] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.451085 31310 ts_tablet_manager.cc:1875] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.453450 31306 ts_tablet_manager.cc:1888] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.453698 31306 log.cc:1196] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/4d05336747e94520bccc53cdb85e51b7
I20240503 14:13:27.455526 31310 ts_tablet_manager.cc:1888] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.455643 31310 log.cc:1196] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/ef0b6bb524e842f5ade540e691e50a97
I20240503 14:13:27.456024 31310 ts_tablet_manager.cc:1909] T ef0b6bb524e842f5ade540e691e50a97 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.455819 31306 ts_tablet_manager.cc:1909] T 4d05336747e94520bccc53cdb85e51b7 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:27.456887 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet ef0b6bb524e842f5ade540e691e50a97 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.457551 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 4d05336747e94520bccc53cdb85e51b7 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.458106 31306 tablet_replica.cc:330] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:27.458396 31306 raft_consensus.cc:2229] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.458685 31306 raft_consensus.cc:2258] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.459594 31306 ts_tablet_manager.cc:1875] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.467981 31306 ts_tablet_manager.cc:1888] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.468176 31306 log.cc:1196] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/07f07ce659e844e3afc090a0753c1128
I20240503 14:13:27.468773 31306 ts_tablet_manager.cc:1909] T 07f07ce659e844e3afc090a0753c1128 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:27.469744 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 07f07ce659e844e3afc090a0753c1128 (table rebalance_test_table_2 [id=830ef3879f3f4b359784c0aa6890ebfe]) successfully deleted
I20240503 14:13:27.588501 31203 raft_consensus.cc:3037] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.591872 31339 consensus_queue.cc:1035] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.598219 31339 consensus_queue.cc:1035] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.700839 31339 raft_consensus.cc:482] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.701035 31339 raft_consensus.cc:504] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.701597 31339 leader_election.cc:288] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:27.702086 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ae0dc65b694823939ced26668ae1c6" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:27.702067 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ae0dc65b694823939ced26668ae1c6" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:27.702311 30809 raft_consensus.cc:2454] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:27.702306 31203 raft_consensus.cc:2454] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:27.702699 30875 leader_election.cc:302] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:27.702975 31339 raft_consensus.cc:2781] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.703074 31339 raft_consensus.cc:482] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.703176 31339 raft_consensus.cc:3037] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.704995 31339 raft_consensus.cc:504] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.705576 31339 leader_election.cc:288] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:27.705946 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ae0dc65b694823939ced26668ae1c6" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:27.706022 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22ae0dc65b694823939ced26668ae1c6" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:27.706147 31203 raft_consensus.cc:3037] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.706221 30809 raft_consensus.cc:3037] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.708909 31203 raft_consensus.cc:2454] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:27.708907 30809 raft_consensus.cc:2454] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:27.709281 30874 leader_election.cc:302] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:27.709538 31339 raft_consensus.cc:2781] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.709704 31339 raft_consensus.cc:686] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:27.709926 31339 consensus_queue.cc:237] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:27.712199 27899 catalog_manager.cc:5484] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:27.731971 30682 consensus_queue.cc:1035] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.737557 31350 consensus_queue.cc:1035] T 54a4dbdf6bd64453a9c2c78dbf550448 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.782097 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:27.790100 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0b457ba3564c52824f84b78cf14c23 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.790648 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0b457ba3564c52824f84b78cf14c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.791021 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 52683e63fe294a15aa301171025afe0a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.791425 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52683e63fe294a15aa301171025afe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.791950 30921 tablet_service.cc:1467] Processing CreateTablet for tablet ff8d8c22d99c462b931a77297c5ff1ff (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.791950 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0b457ba3564c52824f84b78cf14c23 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.792373 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8d8c22d99c462b931a77297c5ff1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.792594 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0b457ba3564c52824f84b78cf14c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.794631 31181 tablet_service.cc:1467] Processing CreateTablet for tablet ff8d8c22d99c462b931a77297c5ff1ff (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.795034 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8d8c22d99c462b931a77297c5ff1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.798555 31320 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:27.798686 28021 tablet_service.cc:1467] Processing CreateTablet for tablet ff8d8c22d99c462b931a77297c5ff1ff (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.798822 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 52683e63fe294a15aa301171025afe0a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.799072 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8d8c22d99c462b931a77297c5ff1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.799216 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52683e63fe294a15aa301171025afe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.800060 31320 tablet_bootstrap.cc:654] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.802433 31266 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:27.803998 31266 tablet_bootstrap.cc:654] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.804361 31320 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:27.804486 31320 ts_tablet_manager.cc:1367] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:27.805250 31406 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:27.805255 31320 raft_consensus.cc:348] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.805455 31320 raft_consensus.cc:374] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.805544 31320 raft_consensus.cc:729] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.805624 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 2d0b457ba3564c52824f84b78cf14c23 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.805769 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 52683e63fe294a15aa301171025afe0a (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.805794 31320 consensus_queue.cc:260] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.806017 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0b457ba3564c52824f84b78cf14c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.806607 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52683e63fe294a15aa301171025afe0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.806927 31406 tablet_bootstrap.cc:654] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.807731 31320 ts_tablet_manager.cc:1398] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:27.808912 31406 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:27.809033 31406 ts_tablet_manager.cc:1367] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:27.810187 31406 raft_consensus.cc:348] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.811265 31339 consensus_queue.cc:1035] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.810402 31406 raft_consensus.cc:374] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.812016 31406 raft_consensus.cc:729] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.812232 31406 consensus_queue.cc:260] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.812772 31406 ts_tablet_manager.cc:1398] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:27.817762 31267 tablet_bootstrap.cc:492] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:27.819535 31267 tablet_bootstrap.cc:654] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.820952 31320 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:27.822352 31267 tablet_bootstrap.cc:492] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:27.822546 31267 ts_tablet_manager.cc:1367] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:27.822757 31320 tablet_bootstrap.cc:654] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.823450 31267 raft_consensus.cc:348] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.823745 31267 raft_consensus.cc:374] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.823824 31267 raft_consensus.cc:729] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.824105 31267 consensus_queue.cc:260] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.825110 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:27.825332 31267 ts_tablet_manager.cc:1398] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:27.833681 30809 raft_consensus.cc:1261] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:27.834590 31270 consensus_queue.cc:1035] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.834813 31203 raft_consensus.cc:1261] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:27.836768 31270 consensus_queue.cc:1035] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:27.844702 28089 tablet.cc:2369] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:27.845722 31409 tablet_bootstrap.cc:492] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:27.847951 31266 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:27.848129 31266 ts_tablet_manager.cc:1367] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.004s
I20240503 14:13:27.848901 31266 raft_consensus.cc:348] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.849144 31266 raft_consensus.cc:374] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.849226 31266 raft_consensus.cc:729] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.849474 31266 consensus_queue.cc:260] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.850446 31266 ts_tablet_manager.cc:1398] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:27.850740 31266 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:27.860306 27894 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 22ae0dc65b694823939ced26668ae1c6 (table rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45]). Expected version 1 got 0
I20240503 14:13:27.861449 31339 consensus_queue.cc:1035] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.862488 31412 mvcc.cc:205] Tried to move back new op lower bound from 7023598009679638528 to 7023598009181982720. Current Snapshot: MvccSnapshot[applied={T|T < 7023598009679638528}]
I20240503 14:13:27.871851 31266 tablet_bootstrap.cc:654] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.872298 31416 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:27.872721 30638 raft_consensus.cc:482] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.872905 31409 tablet_bootstrap.cc:654] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.872895 30638 raft_consensus.cc:504] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.873121 31416 tablet.cc:1620] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:27.873940 30638 leader_election.cc:288] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:27.885802 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45] alter complete (version 1)
I20240503 14:13:27.886119 27899 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 22ae0dc65b694823939ced26668ae1c6 (table rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45]). Expected version 1 got 0
I20240503 14:13:27.878899 31418 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:27.889187 31418 tablet.cc:1620] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:27.892616 31339 raft_consensus.cc:482] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.892769 31339 raft_consensus.cc:504] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.893637 31339 leader_election.cc:288] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:27.893846 31409 tablet_bootstrap.cc:492] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:27.894019 31409 ts_tablet_manager.cc:1367] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.002s
I20240503 14:13:27.894331 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b457ba3564c52824f84b78cf14c23" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:27.894551 31203 raft_consensus.cc:2454] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:27.895011 30875 leader_election.cc:302] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:27.895465 31339 raft_consensus.cc:2781] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.895591 31339 raft_consensus.cc:482] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.895690 31339 raft_consensus.cc:3037] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.898291 31320 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:27.898455 31320 ts_tablet_manager.cc:1367] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.000s
I20240503 14:13:27.899160 31320 raft_consensus.cc:348] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.899370 31320 raft_consensus.cc:374] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.899454 31320 raft_consensus.cc:729] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.899847 31320 consensus_queue.cc:260] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.901434 31320 ts_tablet_manager.cc:1398] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:27.901964 31320 tablet_bootstrap.cc:492] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:27.902333 31409 raft_consensus.cc:348] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.903209 31339 raft_consensus.cc:504] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.905963 31339 leader_election.cc:288] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:27.905822 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b457ba3564c52824f84b78cf14c23" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:27.906381 31202 raft_consensus.cc:3037] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.906422 31409 raft_consensus.cc:374] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.906533 31409 raft_consensus.cc:729] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.906832 31409 consensus_queue.cc:260] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.907358 31409 ts_tablet_manager.cc:1398] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:27.908636 31266 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:27.908767 31266 ts_tablet_manager.cc:1367] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.058s user 0.001s sys 0.005s
I20240503 14:13:27.909103 31202 raft_consensus.cc:2454] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:27.909639 30875 leader_election.cc:302] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:27.909577 31266 raft_consensus.cc:348] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.909804 31266 raft_consensus.cc:374] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.910104 31339 raft_consensus.cc:2781] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.909904 31266 raft_consensus.cc:729] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.910382 31339 raft_consensus.cc:686] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:27.910606 31266 consensus_queue.cc:260] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.911285 31266 ts_tablet_manager.cc:1398] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:13:27.910670 31339 consensus_queue.cc:237] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.912559 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 14:13:27.912788 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 14:13:27.913328 31320 tablet_bootstrap.cc:654] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.915307 31409 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:27.919112 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 9c670ee10e964df492f103cfbf1ccad8
I20240503 14:13:27.919920 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c670ee10e964df492f103cfbf1ccad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:27.920145 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 1a5cc9c71a534b7d8c1180b6e1d69af9
I20240503 14:13:27.920236 31311 tablet_replica.cc:330] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:27.920423 31311 raft_consensus.cc:2229] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.920678 31311 raft_consensus.cc:2258] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.921255 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:27.921324 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 07bd28e4bc574526a170efca0060ea07
I20240503 14:13:27.921407 31311 ts_tablet_manager.cc:1875] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.924239 31409 tablet_bootstrap.cc:654] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.924464 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet 07bd28e4bc574526a170efca0060ea07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:27.924742 31312 tablet_replica.cc:330] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:27.924795 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c670ee10e964df492f103cfbf1ccad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:27.924973 31312 raft_consensus.cc:2229] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.925216 31312 raft_consensus.cc:2258] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.925285 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:27.925413 28413 tablet_service.cc:1514] Processing DeleteTablet for tablet 07bd28e4bc574526a170efca0060ea07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:27.925551 31306 tablet_replica.cc:330] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:27.925776 31306 raft_consensus.cc:2229] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.925937 31312 ts_tablet_manager.cc:1875] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.926136 31306 raft_consensus.cc:2258] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.926889 31306 ts_tablet_manager.cc:1875] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:13:27.928543 28370 consensus_peers.cc:485] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:27.931429 31306 ts_tablet_manager.cc:1888] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.931560 31306 log.cc:1196] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/1a5cc9c71a534b7d8c1180b6e1d69af9
I20240503 14:13:27.931891 31306 ts_tablet_manager.cc:1909] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:27.932220 31312 ts_tablet_manager.cc:1888] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.932372 31312 log.cc:1196] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/07bd28e4bc574526a170efca0060ea07
I20240503 14:13:27.932684 31306 tablet_replica.cc:330] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:27.932780 31312 ts_tablet_manager.cc:1909] T 07bd28e4bc574526a170efca0060ea07 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:27.932771 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.932969 31306 raft_consensus.cc:2229] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.933269 31306 raft_consensus.cc:2258] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.933660 31312 tablet_replica.cc:330] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:27.933701 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 07bd28e4bc574526a170efca0060ea07 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.933862 31306 ts_tablet_manager.cc:1875] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.933861 31312 raft_consensus.cc:2229] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.934111 31312 raft_consensus.cc:2258] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.934780 31312 ts_tablet_manager.cc:1875] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.935887 27894 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 22ae0dc65b694823939ced26668ae1c6 (table rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45]). Expected version 1 got 0
I20240503 14:13:27.938771 31306 ts_tablet_manager.cc:1888] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.938993 31306 log.cc:1196] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/07bd28e4bc574526a170efca0060ea07
I20240503 14:13:27.939606 31306 ts_tablet_manager.cc:1909] T 07bd28e4bc574526a170efca0060ea07 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:27.940333 31312 ts_tablet_manager.cc:1888] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.940505 31312 log.cc:1196] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c670ee10e964df492f103cfbf1ccad8
I20240503 14:13:27.940824 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 07bd28e4bc574526a170efca0060ea07 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.940956 31312 ts_tablet_manager.cc:1909] T 9c670ee10e964df492f103cfbf1ccad8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:27.941532 31311 ts_tablet_manager.cc:1888] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.941664 31311 log.cc:1196] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c670ee10e964df492f103cfbf1ccad8
I20240503 14:13:27.941893 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 9c670ee10e964df492f103cfbf1ccad8 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.942087 31311 ts_tablet_manager.cc:1909] T 9c670ee10e964df492f103cfbf1ccad8 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:27.942970 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 9c670ee10e964df492f103cfbf1ccad8 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.944921 31311 tablet_replica.cc:330] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:27.945195 31311 raft_consensus.cc:2229] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.945197 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c670ee10e964df492f103cfbf1ccad8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:27.945427 31311 raft_consensus.cc:2258] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.945571 31310 tablet_replica.cc:330] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.945766 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:27.945791 31310 raft_consensus.cc:2229] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.946147 31310 raft_consensus.cc:2258] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.946377 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 07bd28e4bc574526a170efca0060ea07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:27.946866 31310 ts_tablet_manager.cc:1875] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.947567 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d8c22d99c462b931a77297c5ff1ff" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:27.947837 31202 raft_consensus.cc:2454] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:27.950171 31311 ts_tablet_manager.cc:1875] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.951910 27894 catalog_manager.cc:5484] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:27.953037 31310 ts_tablet_manager.cc:1888] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.953498 27977 leader_election.cc:302] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:27.953256 31310 log.cc:1196] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/9c670ee10e964df492f103cfbf1ccad8
I20240503 14:13:27.954097 31310 ts_tablet_manager.cc:1909] T 9c670ee10e964df492f103cfbf1ccad8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.954875 31316 raft_consensus.cc:2781] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.954968 31316 raft_consensus.cc:482] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.955087 31316 raft_consensus.cc:3037] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.957635 31310 tablet_replica.cc:330] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.957880 31310 raft_consensus.cc:2229] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.958029 31316 raft_consensus.cc:504] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.958185 31310 raft_consensus.cc:2258] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.958693 31316 leader_election.cc:288] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:27.958956 31310 ts_tablet_manager.cc:1875] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.960858 31320 tablet_bootstrap.cc:492] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:27.961961 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 9c670ee10e964df492f103cfbf1ccad8 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.961880 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d8c22d99c462b931a77297c5ff1ff" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:27.962304 31320 ts_tablet_manager.cc:1367] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20240503 14:13:27.962363 30941 raft_consensus.cc:2454] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:27.963196 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d8c22d99c462b931a77297c5ff1ff" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:27.963433 31202 raft_consensus.cc:3037] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.967032 31202 raft_consensus.cc:2454] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:27.968108 27977 leader_election.cc:302] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:27.968348 31316 raft_consensus.cc:2781] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.968752 31316 raft_consensus.cc:686] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:27.968991 31316 consensus_queue.cc:237] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.970322 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d8c22d99c462b931a77297c5ff1ff" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:27.970528 30941 raft_consensus.cc:3037] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.970645 31409 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:27.970826 31409 ts_tablet_manager.cc:1367] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.000s
I20240503 14:13:27.970300 31320 raft_consensus.cc:348] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.971773 31320 raft_consensus.cc:374] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.971693 31409 raft_consensus.cc:348] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.971889 31320 raft_consensus.cc:729] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.971920 31409 raft_consensus.cc:374] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.971994 31409 raft_consensus.cc:729] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.972239 31409 consensus_queue.cc:260] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:27.972786 31409 ts_tablet_manager.cc:1398] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:27.972993 31320 consensus_queue.cc:260] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:27.973678 30941 raft_consensus.cc:2454] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:27.975311 31320 ts_tablet_manager.cc:1398] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:13:27.980883 31311 ts_tablet_manager.cc:1888] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.981436 31431 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:27.982290 31310 ts_tablet_manager.cc:1888] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.982525 31431 tablet.cc:1620] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:27.983203 27894 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 22ae0dc65b694823939ced26668ae1c6 (table rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45]). Expected version 1 got 0
I20240503 14:13:27.985364 31311 log.cc:1196] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a5cc9c71a534b7d8c1180b6e1d69af9
I20240503 14:13:27.985913 31311 ts_tablet_manager.cc:1909] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:27.987166 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.987452 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b457ba3564c52824f84b78cf14c23" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:27.987473 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d0b457ba3564c52824f84b78cf14c23" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:27.987756 31071 raft_consensus.cc:2454] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:27.989898 31310 log.cc:1196] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/1a5cc9c71a534b7d8c1180b6e1d69af9
I20240503 14:13:27.990491 31310 ts_tablet_manager.cc:1909] T 1a5cc9c71a534b7d8c1180b6e1d69af9 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.991441 31310 tablet_replica.cc:330] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:27.991540 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 1a5cc9c71a534b7d8c1180b6e1d69af9 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:27.991688 31310 raft_consensus.cc:2229] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.992267 31310 raft_consensus.cc:2258] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.992936 31310 ts_tablet_manager.cc:1875] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.992790 27894 catalog_manager.cc:5484] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:27.997918 31310 ts_tablet_manager.cc:1888] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:27.998028 31310 log.cc:1196] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/07bd28e4bc574526a170efca0060ea07
I20240503 14:13:27.998390 31310 ts_tablet_manager.cc:1909] T 07bd28e4bc574526a170efca0060ea07 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:27.999141 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 07bd28e4bc574526a170efca0060ea07 (table rebalance_test_table_4 [id=ff4339d2a3f34f80b913a2b5e70d0f19]) successfully deleted
I20240503 14:13:28.001065 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:28.007779 31416 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:28.008352 31416 tablet.cc:1620] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:28.008838 31418 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:28.009284 31418 tablet.cc:1620] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:28.009367 31431 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:28.009801 31431 tablet.cc:1620] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:28.016639 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=89b0d910816144c79e4685fbb09dfd45] alter complete (version 2)
I20240503 14:13:28.087755 31346 raft_consensus.cc:482] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.087927 31346 raft_consensus.cc:504] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:28.088518 31346 leader_election.cc:288] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:28.088945 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52683e63fe294a15aa301171025afe0a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:28.089182 30809 raft_consensus.cc:2454] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:28.089653 31005 leader_election.cc:302] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:28.089941 31346 raft_consensus.cc:2781] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.090049 31346 raft_consensus.cc:482] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.090142 31346 raft_consensus.cc:3037] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.092640 31346 raft_consensus.cc:504] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:28.093398 31346 leader_election.cc:288] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:28.093705 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52683e63fe294a15aa301171025afe0a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:28.093900 30809 raft_consensus.cc:3037] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.095847 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52683e63fe294a15aa301171025afe0a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:28.095927 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52683e63fe294a15aa301171025afe0a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:28.096122 31202 raft_consensus.cc:2454] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:28.096197 30809 raft_consensus.cc:2454] T 52683e63fe294a15aa301171025afe0a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:28.096558 31005 leader_election.cc:302] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters: 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:13:28.096879 31346 raft_consensus.cc:2781] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.098532 31346 raft_consensus.cc:686] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:28.098985 31346 consensus_queue.cc:237] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:28.102046 27896 catalog_manager.cc:5484] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:28.119063 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:28.125356 31416 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:28.125993 31431 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:28.126858 31418 tablet.cc:1722] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:28.133317 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=89b0d910816144c79e4685fbb09dfd45] alter complete (version 3)
I20240503 14:13:28.235878 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:28.236099 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:28.238966 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 22ae0dc65b694823939ced26668ae1c6
I20240503 14:13:28.239830 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 22ae0dc65b694823939ced26668ae1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:28.239945 31182 tablet_service.cc:1514] Processing DeleteTablet for tablet 22ae0dc65b694823939ced26668ae1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:28.240247 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 22ae0dc65b694823939ced26668ae1c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:28.241024 31439 tablet_replica.cc:330] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:28.241307 31439 raft_consensus.cc:2229] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.241595 31439 raft_consensus.cc:2258] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.242856 31439 ts_tablet_manager.cc:1875] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.251250 31439 ts_tablet_manager.cc:1888] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:28.251436 31439 log.cc:1196] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/22ae0dc65b694823939ced26668ae1c6
I20240503 14:13:28.251981 31439 ts_tablet_manager.cc:1909] T 22ae0dc65b694823939ced26668ae1c6 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:28.253000 31437 tablet_replica.cc:330] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:28.253163 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 22ae0dc65b694823939ced26668ae1c6 (table rebalancer_dynamic_table_new_name [id=89b0d910816144c79e4685fbb09dfd45]) successfully deleted
I20240503 14:13:28.253350 31437 raft_consensus.cc:2229] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.253706 31437 raft_consensus.cc:2258] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.254396 31438 tablet_replica.cc:330] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:28.254707 31437 ts_tablet_manager.cc:1875] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.254878 31438 raft_consensus.cc:2229] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.255220 31438 raft_consensus.cc:2258] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.256383 31438 ts_tablet_manager.cc:1875] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.259011 31437 ts_tablet_manager.cc:1888] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:28.259100 31437 log.cc:1196] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/22ae0dc65b694823939ced26668ae1c6
I20240503 14:13:28.259478 31437 ts_tablet_manager.cc:1909] T 22ae0dc65b694823939ced26668ae1c6 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:28.260257 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 22ae0dc65b694823939ced26668ae1c6 (table rebalancer_dynamic_table_new_name [id=89b0d910816144c79e4685fbb09dfd45]) successfully deleted
I20240503 14:13:28.261044 31438 ts_tablet_manager.cc:1888] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:28.261507 31438 log.cc:1196] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/22ae0dc65b694823939ced26668ae1c6
I20240503 14:13:28.261915 31438 ts_tablet_manager.cc:1909] T 22ae0dc65b694823939ced26668ae1c6 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:28.263324 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 22ae0dc65b694823939ced26668ae1c6 (table rebalancer_dynamic_table_new_name [id=89b0d910816144c79e4685fbb09dfd45]) successfully deleted
I20240503 14:13:28.341372 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:28.344396 31072 raft_consensus.cc:3037] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.348196 31339 consensus_queue.cc:1035] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.356444 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7a48c5a122421797e1924fc36f3c80 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21]), partition=
I20240503 14:13:28.356428 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7a48c5a122421797e1924fc36f3c80 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21]), partition=
I20240503 14:13:28.356812 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7a48c5a122421797e1924fc36f3c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.356844 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7a48c5a122421797e1924fc36f3c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.360108 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 6c7a48c5a122421797e1924fc36f3c80 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21]), partition=
I20240503 14:13:28.360414 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c7a48c5a122421797e1924fc36f3c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.360666 31413 consensus_queue.cc:1035] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.365082 31266 tablet_bootstrap.cc:492] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:28.366564 31266 tablet_bootstrap.cc:654] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.370646 31409 tablet_bootstrap.cc:492] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:28.372116 31409 tablet_bootstrap.cc:654] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.376325 31266 tablet_bootstrap.cc:492] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:28.376515 31266 ts_tablet_manager.cc:1367] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:13:28.377539 31266 raft_consensus.cc:348] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.377776 31266 raft_consensus.cc:374] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.377868 31266 raft_consensus.cc:729] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.378161 31266 consensus_queue.cc:260] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.379348 31266 ts_tablet_manager.cc:1398] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:28.379647 31409 tablet_bootstrap.cc:492] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:28.379786 31409 ts_tablet_manager.cc:1367] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 14:13:28.380399 31444 tablet_bootstrap.cc:492] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:28.380563 31409 raft_consensus.cc:348] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.380789 31409 raft_consensus.cc:374] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.380893 31409 raft_consensus.cc:729] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.381136 31409 consensus_queue.cc:260] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.382331 31409 ts_tablet_manager.cc:1398] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:13:28.383908 31444 tablet_bootstrap.cc:654] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.386934 31444 tablet_bootstrap.cc:492] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:28.387109 31444 ts_tablet_manager.cc:1367] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:28.387924 31444 raft_consensus.cc:348] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.388124 31444 raft_consensus.cc:374] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.388221 31444 raft_consensus.cc:729] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.388666 31444 consensus_queue.cc:260] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.389403 31444 ts_tablet_manager.cc:1398] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:28.408344 31316 consensus_queue.cc:1035] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.413479 30638 consensus_queue.cc:1035] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.422892 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 14:13:28.423053 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 14:13:28.426281 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 95e9ade3fd96457ca25cef44a301644b
I20240503 14:13:28.426824 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 5aae2864fae8454ca494b5e4ba47ac5c
I20240503 14:13:28.427222 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e9ade3fd96457ca25cef44a301644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:28.427294 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 457b7d72f79a4431a9442bea4097b2a8
I20240503 14:13:28.427525 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aae2864fae8454ca494b5e4ba47ac5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:28.427533 31310 tablet_replica.cc:330] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:28.427826 28413 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e9ade3fd96457ca25cef44a301644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:28.427990 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aae2864fae8454ca494b5e4ba47ac5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:28.428040 31310 raft_consensus.cc:2229] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.428251 31306 tablet_replica.cc:330] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:28.428267 31310 raft_consensus.cc:2258] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.428483 31306 raft_consensus.cc:2229] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.428797 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aae2864fae8454ca494b5e4ba47ac5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:28.428855 31310 ts_tablet_manager.cc:1875] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.428839 31306 raft_consensus.cc:2258] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.429059 31312 tablet_replica.cc:330] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:28.429301 31312 raft_consensus.cc:2229] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.429509 31312 raft_consensus.cc:2258] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.429620 31306 ts_tablet_manager.cc:1875] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.429678 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 95e9ade3fd96457ca25cef44a301644b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:28.430151 31312 ts_tablet_manager.cc:1875] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.433049 31311 tablet_replica.cc:330] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:28.433192 28153 tablet_service.cc:1514] Processing DeleteTablet for tablet 457b7d72f79a4431a9442bea4097b2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:28.433321 31311 raft_consensus.cc:2229] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.433609 31311 raft_consensus.cc:2258] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.434340 31311 ts_tablet_manager.cc:1875] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.436040 31310 ts_tablet_manager.cc:1888] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.436146 31310 log.cc:1196] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/95e9ade3fd96457ca25cef44a301644b
I20240503 14:13:28.436553 31310 ts_tablet_manager.cc:1909] T 95e9ade3fd96457ca25cef44a301644b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:28.436939 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 457b7d72f79a4431a9442bea4097b2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:28.437112 31312 ts_tablet_manager.cc:1888] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.437426 31312 log.cc:1196] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/5aae2864fae8454ca494b5e4ba47ac5c
I20240503 14:13:28.437469 31310 tablet_replica.cc:330] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:28.437690 31310 raft_consensus.cc:2229] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.437826 31312 ts_tablet_manager.cc:1909] T 5aae2864fae8454ca494b5e4ba47ac5c P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:28.438011 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 95e9ade3fd96457ca25cef44a301644b (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.438030 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 457b7d72f79a4431a9442bea4097b2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:28.438406 31310 raft_consensus.cc:2258] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.438791 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 5aae2864fae8454ca494b5e4ba47ac5c (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.439395 31310 ts_tablet_manager.cc:1875] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.439740 31311 ts_tablet_manager.cc:1888] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.439919 31311 log.cc:1196] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/95e9ade3fd96457ca25cef44a301644b
I20240503 14:13:28.440407 31311 ts_tablet_manager.cc:1909] T 95e9ade3fd96457ca25cef44a301644b P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:28.441668 31306 ts_tablet_manager.cc:1888] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.441828 31306 log.cc:1196] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/95e9ade3fd96457ca25cef44a301644b
I20240503 14:13:28.442045 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 95e9ade3fd96457ca25cef44a301644b (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.442313 31306 ts_tablet_manager.cc:1909] T 95e9ade3fd96457ca25cef44a301644b P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:28.442452 31311 tablet_replica.cc:330] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:28.442726 31311 raft_consensus.cc:2229] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.442996 31311 raft_consensus.cc:2258] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.443451 31306 tablet_replica.cc:330] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:28.443491 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 95e9ade3fd96457ca25cef44a301644b (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.443665 31306 raft_consensus.cc:2229] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.443766 31311 ts_tablet_manager.cc:1875] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.443922 31306 raft_consensus.cc:2258] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.444011 31310 ts_tablet_manager.cc:1888] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.444294 31310 log.cc:1196] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/5aae2864fae8454ca494b5e4ba47ac5c
I20240503 14:13:28.444631 31310 ts_tablet_manager.cc:1909] T 5aae2864fae8454ca494b5e4ba47ac5c P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:28.444690 31306 ts_tablet_manager.cc:1875] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.445389 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 5aae2864fae8454ca494b5e4ba47ac5c (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.445581 31310 tablet_replica.cc:330] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:28.445737 31310 raft_consensus.cc:2229] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.445932 31310 raft_consensus.cc:2258] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.446516 31310 ts_tablet_manager.cc:1875] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.451201 31310 ts_tablet_manager.cc:1888] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.451309 31310 log.cc:1196] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/457b7d72f79a4431a9442bea4097b2a8
I20240503 14:13:28.451457 31311 ts_tablet_manager.cc:1888] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.451625 31311 log.cc:1196] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/457b7d72f79a4431a9442bea4097b2a8
I20240503 14:13:28.451745 31310 ts_tablet_manager.cc:1909] T 457b7d72f79a4431a9442bea4097b2a8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:28.452347 31306 ts_tablet_manager.cc:1888] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.452107 31311 ts_tablet_manager.cc:1909] T 457b7d72f79a4431a9442bea4097b2a8 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:28.452507 31306 log.cc:1196] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/5aae2864fae8454ca494b5e4ba47ac5c
I20240503 14:13:28.452656 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 457b7d72f79a4431a9442bea4097b2a8 (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.452944 31306 ts_tablet_manager.cc:1909] T 5aae2864fae8454ca494b5e4ba47ac5c P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:28.453545 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 457b7d72f79a4431a9442bea4097b2a8 (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.453919 31306 tablet_replica.cc:330] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:28.453955 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 5aae2864fae8454ca494b5e4ba47ac5c (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.454141 31306 raft_consensus.cc:2229] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.454466 31306 raft_consensus.cc:2258] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.455217 31306 ts_tablet_manager.cc:1875] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.459508 31306 ts_tablet_manager.cc:1888] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.459707 31306 log.cc:1196] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/457b7d72f79a4431a9442bea4097b2a8
I20240503 14:13:28.460135 31306 ts_tablet_manager.cc:1909] T 457b7d72f79a4431a9442bea4097b2a8 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:28.460899 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 457b7d72f79a4431a9442bea4097b2a8 (table rebalance_test_table_6 [id=cdfef7740e5346dab2e7046408a05b4f]) successfully deleted
I20240503 14:13:28.488320 31346 consensus_queue.cc:1035] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.492689 31202 raft_consensus.cc:3037] T 52683e63fe294a15aa301171025afe0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.495563 31346 consensus_queue.cc:1035] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.624770 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:28.633942 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 5608b8d520274a19929fb3f793ef0b84 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.634380 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 5608b8d520274a19929fb3f793ef0b84 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.634570 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5608b8d520274a19929fb3f793ef0b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.634653 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 5608b8d520274a19929fb3f793ef0b84 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.634802 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5608b8d520274a19929fb3f793ef0b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.635126 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5608b8d520274a19929fb3f793ef0b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.638612 30921 tablet_service.cc:1467] Processing CreateTablet for tablet bb1ec320c4114eec8915b7f2bcffc2c3 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.638746 31051 tablet_service.cc:1467] Processing CreateTablet for tablet fbd2cac7ae8e45b6bd407b119efe3a51 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.639016 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1ec320c4114eec8915b7f2bcffc2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.638945 31052 tablet_service.cc:1467] Processing CreateTablet for tablet bb1ec320c4114eec8915b7f2bcffc2c3 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.639140 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd2cac7ae8e45b6bd407b119efe3a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.639369 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1ec320c4114eec8915b7f2bcffc2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.643762 28154 tablet_service.cc:1467] Processing CreateTablet for tablet bb1ec320c4114eec8915b7f2bcffc2c3 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.644172 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1ec320c4114eec8915b7f2bcffc2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.647977 28023 tablet_service.cc:1467] Processing CreateTablet for tablet fbd2cac7ae8e45b6bd407b119efe3a51 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.648012 30790 tablet_service.cc:1467] Processing CreateTablet for tablet fbd2cac7ae8e45b6bd407b119efe3a51 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.648388 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd2cac7ae8e45b6bd407b119efe3a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.648399 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd2cac7ae8e45b6bd407b119efe3a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.650491 31266 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:28.651989 31266 tablet_bootstrap.cc:654] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.653626 31454 tablet_bootstrap.cc:492] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:28.655164 31454 tablet_bootstrap.cc:654] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.657750 31444 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:28.658509 31409 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:28.659251 31444 tablet_bootstrap.cc:654] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.660527 31409 tablet_bootstrap.cc:654] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.673682 31444 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:28.673714 31266 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:28.673885 31266 ts_tablet_manager.cc:1367] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.005s
I20240503 14:13:28.674029 31409 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:28.674142 31444 ts_tablet_manager.cc:1367] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240503 14:13:28.674286 31409 ts_tablet_manager.cc:1367] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240503 14:13:28.674718 31266 raft_consensus.cc:348] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.674957 31266 raft_consensus.cc:374] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.675077 31266 raft_consensus.cc:729] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.675315 31266 consensus_queue.cc:260] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.675396 31444 raft_consensus.cc:348] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.675755 31444 raft_consensus.cc:374] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.675868 31444 raft_consensus.cc:729] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.675942 31266 ts_tablet_manager.cc:1398] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:28.676191 31444 consensus_queue.cc:260] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.675189 31409 raft_consensus.cc:348] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.676517 31409 raft_consensus.cc:374] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.676617 31409 raft_consensus.cc:729] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.676934 31409 consensus_queue.cc:260] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.677464 31409 ts_tablet_manager.cc:1398] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:28.677958 31460 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:28.679519 31460 tablet_bootstrap.cc:654] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.680821 31455 tablet_bootstrap.cc:492] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:28.682917 31409 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:28.682910 31453 tablet_bootstrap.cc:492] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:28.683822 31455 tablet_bootstrap.cc:654] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.684526 31409 tablet_bootstrap.cc:654] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.684526 31453 tablet_bootstrap.cc:654] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.685730 31444 ts_tablet_manager.cc:1398] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:28.686534 31454 tablet_bootstrap.cc:492] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:28.687263 31454 ts_tablet_manager.cc:1367] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.008s
I20240503 14:13:28.688074 31454 raft_consensus.cc:348] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.688347 31454 raft_consensus.cc:374] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.688510 31454 raft_consensus.cc:729] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.688768 31454 consensus_queue.cc:260] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.689849 31460 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:28.690027 31460 ts_tablet_manager.cc:1367] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:13:28.690034 31454 ts_tablet_manager.cc:1398] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:28.691022 31460 raft_consensus.cc:348] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.691346 31460 raft_consensus.cc:374] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.691565 31460 raft_consensus.cc:729] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.691884 31409 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:28.691836 31460 consensus_queue.cc:260] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.692118 31409 ts_tablet_manager.cc:1367] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240503 14:13:28.692903 31409 raft_consensus.cc:348] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.693416 31409 raft_consensus.cc:374] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.693543 31409 raft_consensus.cc:729] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.693802 31409 consensus_queue.cc:260] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.694037 31455 tablet_bootstrap.cc:492] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:28.694228 31455 ts_tablet_manager.cc:1367] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 14:13:28.695220 31409 ts_tablet_manager.cc:1398] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:28.696449 31460 ts_tablet_manager.cc:1398] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:28.696768 31455 raft_consensus.cc:348] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.699636 31455 raft_consensus.cc:374] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.699753 31455 raft_consensus.cc:729] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.699975 31455 consensus_queue.cc:260] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.700487 31455 ts_tablet_manager.cc:1398] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:13:28.701045 31455 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:28.701543 31453 tablet_bootstrap.cc:492] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:28.701699 31453 ts_tablet_manager.cc:1367] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 14:13:28.702494 31453 raft_consensus.cc:348] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.702669 31455 tablet_bootstrap.cc:654] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.702721 31453 raft_consensus.cc:374] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.702812 31453 raft_consensus.cc:729] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.703058 31453 consensus_queue.cc:260] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.703672 31453 ts_tablet_manager.cc:1398] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:28.704470 31455 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:28.704614 31455 ts_tablet_manager.cc:1367] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:28.705207 31455 raft_consensus.cc:348] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.705463 31455 raft_consensus.cc:374] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.705541 31455 raft_consensus.cc:729] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.705763 31455 consensus_queue.cc:260] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.706207 31455 ts_tablet_manager.cc:1398] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:28.743453 31452 raft_consensus.cc:482] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.743623 31452 raft_consensus.cc:504] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.744282 31452 leader_election.cc:288] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:28.744853 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5608b8d520274a19929fb3f793ef0b84" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:28.745088 28432 raft_consensus.cc:2454] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:28.745492 28110 leader_election.cc:302] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:28.745781 31452 raft_consensus.cc:2781] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.745896 31452 raft_consensus.cc:482] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.745993 31452 raft_consensus.cc:3037] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.748517 31452 raft_consensus.cc:504] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.749081 31452 leader_election.cc:288] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:28.749528 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5608b8d520274a19929fb3f793ef0b84" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:28.749693 28432 raft_consensus.cc:3037] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.751348 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5608b8d520274a19929fb3f793ef0b84" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:28.751355 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5608b8d520274a19929fb3f793ef0b84" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:28.751597 31202 raft_consensus.cc:2454] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:28.751777 28432 raft_consensus.cc:2454] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:28.752192 28110 leader_election.cc:302] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters: 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:13:28.752449 31452 raft_consensus.cc:2781] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.752564 31452 raft_consensus.cc:686] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:28.752800 31452 consensus_queue.cc:237] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:28.755775 27896 catalog_manager.cc:5484] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:28.757292 31271 raft_consensus.cc:482] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.757479 31271 raft_consensus.cc:504] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.759030 31271 leader_election.cc:288] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:28.765290 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7a48c5a122421797e1924fc36f3c80" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:28.765486 31072 raft_consensus.cc:2454] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:28.765656 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7a48c5a122421797e1924fc36f3c80" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:28.765921 30941 raft_consensus.cc:2454] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:28.765998 30745 leader_election.cc:302] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:28.766414 31271 raft_consensus.cc:2781] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.766559 31271 raft_consensus.cc:482] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.766659 31271 raft_consensus.cc:3037] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.768848 31271 raft_consensus.cc:504] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.769354 31271 leader_election.cc:288] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:28.769737 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7a48c5a122421797e1924fc36f3c80" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:28.769804 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c7a48c5a122421797e1924fc36f3c80" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:28.769898 31072 raft_consensus.cc:3037] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.769996 30941 raft_consensus.cc:3037] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.771754 31072 raft_consensus.cc:2454] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:28.771811 30941 raft_consensus.cc:2454] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:28.772166 30745 leader_election.cc:302] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:28.772431 31271 raft_consensus.cc:2781] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.772858 31271 raft_consensus.cc:686] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:28.773255 31271 consensus_queue.cc:237] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:28.776393 27896 catalog_manager.cc:5484] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:28.811625 31413 raft_consensus.cc:482] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.811796 31413 raft_consensus.cc:504] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.812484 31413 leader_election.cc:288] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:28.812899 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1ec320c4114eec8915b7f2bcffc2c3" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:28.813261 31072 raft_consensus.cc:2454] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:28.813737 30876 leader_election.cc:302] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:28.814014 31413 raft_consensus.cc:2781] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.814114 31413 raft_consensus.cc:482] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.814260 31413 raft_consensus.cc:3037] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.816258 31413 raft_consensus.cc:504] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.817654 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1ec320c4114eec8915b7f2bcffc2c3" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:28.817885 31072 raft_consensus.cc:3037] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.819921 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1ec320c4114eec8915b7f2bcffc2c3" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:28.819938 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1ec320c4114eec8915b7f2bcffc2c3" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:28.820180 28172 raft_consensus.cc:2454] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:28.820289 31072 raft_consensus.cc:2454] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:28.820659 30876 leader_election.cc:302] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters: 6409961ee8c0445687c11724213c44a9
I20240503 14:13:28.820894 31413 leader_election.cc:288] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:28.820983 31339 raft_consensus.cc:2781] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.821352 31339 raft_consensus.cc:686] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:28.821599 31339 consensus_queue.cc:237] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:28.824244 27896 catalog_manager.cc:5484] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:28.832574 30638 raft_consensus.cc:482] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.832711 30638 raft_consensus.cc:504] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.833443 30638 leader_election.cc:288] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
W20240503 14:13:28.838383 28220 tablet.cc:2369] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:28.842445 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd2cac7ae8e45b6bd407b119efe3a51" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:28.842523 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd2cac7ae8e45b6bd407b119efe3a51" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:28.842646 31072 raft_consensus.cc:2454] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:28.842761 30810 raft_consensus.cc:2454] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:28.843070 27978 leader_election.cc:302] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:28.843370 30638 raft_consensus.cc:2781] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.843508 30638 raft_consensus.cc:482] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.843616 30638 raft_consensus.cc:3037] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.845755 30638 raft_consensus.cc:504] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.846426 30638 leader_election.cc:288] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:28.847245 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd2cac7ae8e45b6bd407b119efe3a51" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:28.847441 31072 raft_consensus.cc:3037] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.849407 31072 raft_consensus.cc:2454] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:28.849730 27978 leader_election.cc:302] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:28.850095 31316 raft_consensus.cc:2781] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.850171 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd2cac7ae8e45b6bd407b119efe3a51" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:28.850275 31316 raft_consensus.cc:686] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:28.850401 30810 raft_consensus.cc:3037] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.850555 31316 consensus_queue.cc:237] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:28.858497 30810 raft_consensus.cc:2454] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:28.869874 27896 catalog_manager.cc:5484] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:28.927868 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240503 14:13:28.929013 28480 tablet.cc:2369] T 5608b8d520274a19929fb3f793ef0b84 P 5388902973134958a030fd63ab4faba4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:28.934454 30941 raft_consensus.cc:1261] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:28.934456 31072 raft_consensus.cc:1261] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:28.935267 31271 consensus_queue.cc:1035] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.935665 31473 consensus_queue.cc:1035] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.938365 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 14:13:28.938552 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 14:13:28.940654 27894 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 6c7a48c5a122421797e1924fc36f3c80 (table rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21]). Expected version 1 got 0
I20240503 14:13:28.941655 31398 mvcc.cc:205] Tried to move back new op lower bound from 7023598014188806144 to 7023598013538267136. Current Snapshot: MvccSnapshot[applied={T|T < 7023598014188806144}]
I20240503 14:13:28.941849 31486 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:28.942607 31486 tablet.cc:1620] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:28.944079 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet b4a09a046a5043a59c50c359304caa85
I20240503 14:13:28.944029 31487 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:28.944520 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 99896ce4c4ec4384a84e4b07d3a19b2d
I20240503 14:13:28.944763 31487 tablet.cc:1620] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:28.945354 27894 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 6c7a48c5a122421797e1924fc36f3c80 (table rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21]). Expected version 1 got 0
I20240503 14:13:28.946010 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a09a046a5043a59c50c359304caa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:28.946216 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet e90f07adac794442be03689f97bfe60a
I20240503 14:13:28.946568 31488 tablet_replica.cc:330] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:28.946781 31488 raft_consensus.cc:2229] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.947068 28153 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a09a046a5043a59c50c359304caa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:28.947275 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 99896ce4c4ec4384a84e4b07d3a19b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:28.947362 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet e90f07adac794442be03689f97bfe60a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:28.947556 31311 tablet_replica.cc:330] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:28.947517 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 99896ce4c4ec4384a84e4b07d3a19b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:28.947755 31311 raft_consensus.cc:2229] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.947784 31306 tablet_replica.cc:330] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:28.947952 31306 raft_consensus.cc:2229] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.947973 31311 raft_consensus.cc:2258] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.948202 31306 raft_consensus.cc:2258] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.948652 31311 ts_tablet_manager.cc:1875] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.948824 31306 ts_tablet_manager.cc:1875] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.950125 31441 mvcc.cc:205] Tried to move back new op lower bound from 7023598014188806144 to 7023598013538267136. Current Snapshot: MvccSnapshot[applied={T|T < 7023598014188806144}]
I20240503 14:13:28.950395 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet e90f07adac794442be03689f97bfe60a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:28.950487 28022 tablet_service.cc:1514] Processing DeleteTablet for tablet e90f07adac794442be03689f97bfe60a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:28.950675 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 99896ce4c4ec4384a84e4b07d3a19b2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:28.950773 31310 tablet_replica.cc:330] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:28.951018 31310 raft_consensus.cc:2229] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.951138 27896 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 6c7a48c5a122421797e1924fc36f3c80 (table rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21]). Expected version 1 got 0
I20240503 14:13:28.951254 31310 raft_consensus.cc:2258] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.952021 31310 ts_tablet_manager.cc:1875] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.952211 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet b4a09a046a5043a59c50c359304caa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:28.956947 31306 ts_tablet_manager.cc:1888] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.957116 31306 log.cc:1196] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/99896ce4c4ec4384a84e4b07d3a19b2d
I20240503 14:13:28.957644 31306 ts_tablet_manager.cc:1909] T 99896ce4c4ec4384a84e4b07d3a19b2d P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:28.958660 31306 tablet_replica.cc:330] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:28.958817 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 99896ce4c4ec4384a84e4b07d3a19b2d (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.959406 31306 raft_consensus.cc:2229] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.959556 31310 ts_tablet_manager.cc:1888] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.959872 31306 raft_consensus.cc:2258] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.960572 31306 ts_tablet_manager.cc:1875] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.960878 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21] alter complete (version 1)
I20240503 14:13:28.963785 31310 log.cc:1196] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/e90f07adac794442be03689f97bfe60a
I20240503 14:13:28.964313 31310 ts_tablet_manager.cc:1909] T e90f07adac794442be03689f97bfe60a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:28.964772 31311 ts_tablet_manager.cc:1888] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.964890 31311 log.cc:1196] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/b4a09a046a5043a59c50c359304caa85
I20240503 14:13:28.965176 31310 tablet_replica.cc:330] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:28.965271 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet e90f07adac794442be03689f97bfe60a (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.965370 31310 raft_consensus.cc:2229] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.965655 31310 raft_consensus.cc:2258] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.966131 31311 ts_tablet_manager.cc:1909] T b4a09a046a5043a59c50c359304caa85 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:28.966277 31310 ts_tablet_manager.cc:1875] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.966286 31306 ts_tablet_manager.cc:1888] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.966418 31306 log.cc:1196] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/e90f07adac794442be03689f97bfe60a
I20240503 14:13:28.966848 31306 ts_tablet_manager.cc:1909] T e90f07adac794442be03689f97bfe60a P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:28.967559 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet b4a09a046a5043a59c50c359304caa85 (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.967654 31311 tablet_replica.cc:330] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:28.967705 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet e90f07adac794442be03689f97bfe60a (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.967850 31311 raft_consensus.cc:2229] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.967948 31488 raft_consensus.cc:2258] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.968070 31311 raft_consensus.cc:2258] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.968677 31311 ts_tablet_manager.cc:1875] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.968663 31488 ts_tablet_manager.cc:1875] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.975085 31489 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:28.976393 27894 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 6c7a48c5a122421797e1924fc36f3c80 (table rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21]). Expected version 1 got 0
I20240503 14:13:28.977113 31488 ts_tablet_manager.cc:1888] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.977167 31489 tablet.cc:1620] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:28.977238 31488 log.cc:1196] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/b4a09a046a5043a59c50c359304caa85
I20240503 14:13:28.977239 31310 ts_tablet_manager.cc:1888] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.977409 31310 log.cc:1196] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/99896ce4c4ec4384a84e4b07d3a19b2d
I20240503 14:13:28.977681 31488 ts_tablet_manager.cc:1909] T b4a09a046a5043a59c50c359304caa85 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:28.978664 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet b4a09a046a5043a59c50c359304caa85 (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.979017 31488 tablet_replica.cc:330] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:28.979030 31310 ts_tablet_manager.cc:1909] T 99896ce4c4ec4384a84e4b07d3a19b2d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:28.979246 31488 raft_consensus.cc:2229] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.979526 31488 raft_consensus.cc:2258] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.979759 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 99896ce4c4ec4384a84e4b07d3a19b2d (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.979847 31310 tablet_replica.cc:330] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:28.980075 31310 raft_consensus.cc:2229] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.980171 31488 ts_tablet_manager.cc:1875] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.980326 31310 raft_consensus.cc:2258] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.980801 31310 ts_tablet_manager.cc:1875] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.981530 31311 ts_tablet_manager.cc:1888] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.981664 31311 log.cc:1196] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/99896ce4c4ec4384a84e4b07d3a19b2d
I20240503 14:13:28.982043 31311 ts_tablet_manager.cc:1909] T 99896ce4c4ec4384a84e4b07d3a19b2d P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:28.982945 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 99896ce4c4ec4384a84e4b07d3a19b2d (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.985172 31310 ts_tablet_manager.cc:1888] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.985282 31310 log.cc:1196] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/b4a09a046a5043a59c50c359304caa85
I20240503 14:13:28.985600 31310 ts_tablet_manager.cc:1909] T b4a09a046a5043a59c50c359304caa85 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:28.986550 31488 ts_tablet_manager.cc:1888] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:28.986738 31488 log.cc:1196] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/e90f07adac794442be03689f97bfe60a
I20240503 14:13:28.986843 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet b4a09a046a5043a59c50c359304caa85 (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:28.987103 31488 ts_tablet_manager.cc:1909] T e90f07adac794442be03689f97bfe60a P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:28.987910 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet e90f07adac794442be03689f97bfe60a (table rebalance_test_table_8 [id=2a70da6f1fa944f7b4b3686765855385]) successfully deleted
I20240503 14:13:29.063206 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:29.069830 31486 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.070480 31487 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.070935 31486 tablet.cc:1620] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:29.070966 31487 tablet.cc:1620] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:29.070941 31489 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.071352 31489 tablet.cc:1620] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:29.079455 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=2df0556092e945fb9098e7420791ab21] alter complete (version 2)
I20240503 14:13:29.182765 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:29.189841 31486 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:29.191253 31487 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:29.195765 31392 consensus_queue.cc:1035] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.196985 31489 tablet.cc:1722] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:29.199847 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=2df0556092e945fb9098e7420791ab21] alter complete (version 3)
I20240503 14:13:29.208743 31202 raft_consensus.cc:3037] T 5608b8d520274a19929fb3f793ef0b84 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.211211 31392 consensus_queue.cc:1035] T 5608b8d520274a19929fb3f793ef0b84 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.221876 31339 consensus_queue.cc:1035] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.225080 28172 raft_consensus.cc:3037] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.227679 31413 consensus_queue.cc:1035] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.306409 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:29.306685 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:29.309463 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 6c7a48c5a122421797e1924fc36f3c80
I20240503 14:13:29.310364 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7a48c5a122421797e1924fc36f3c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:29.310492 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7a48c5a122421797e1924fc36f3c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:29.310485 31052 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c7a48c5a122421797e1924fc36f3c80 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:29.311064 31499 tablet_replica.cc:330] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:29.311246 31498 tablet_replica.cc:330] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:29.311362 31499 raft_consensus.cc:2229] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.311540 31498 raft_consensus.cc:2229] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:29.311609 31500 tablet_replica.cc:330] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:29.311605 31499 raft_consensus.cc:2258] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.312090 31500 raft_consensus.cc:2229] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.312085 31498 raft_consensus.cc:2258] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.312357 31500 raft_consensus.cc:2258] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.312502 31499 ts_tablet_manager.cc:1875] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.312844 31498 ts_tablet_manager.cc:1875] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.313802 31500 ts_tablet_manager.cc:1875] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.317234 31499 ts_tablet_manager.cc:1888] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:29.317329 31499 log.cc:1196] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/6c7a48c5a122421797e1924fc36f3c80
I20240503 14:13:29.317656 31499 ts_tablet_manager.cc:1909] T 6c7a48c5a122421797e1924fc36f3c80 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:29.318617 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 6c7a48c5a122421797e1924fc36f3c80 (table rebalancer_dynamic_table_new_name [id=2df0556092e945fb9098e7420791ab21]) successfully deleted
I20240503 14:13:29.318812 31498 ts_tablet_manager.cc:1888] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:29.318976 31498 log.cc:1196] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/6c7a48c5a122421797e1924fc36f3c80
I20240503 14:13:29.319414 31498 ts_tablet_manager.cc:1909] T 6c7a48c5a122421797e1924fc36f3c80 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:29.320350 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 6c7a48c5a122421797e1924fc36f3c80 (table rebalancer_dynamic_table_new_name [id=2df0556092e945fb9098e7420791ab21]) successfully deleted
I20240503 14:13:29.320387 31500 ts_tablet_manager.cc:1888] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:29.320493 31500 log.cc:1196] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/6c7a48c5a122421797e1924fc36f3c80
I20240503 14:13:29.320887 31500 ts_tablet_manager.cc:1909] T 6c7a48c5a122421797e1924fc36f3c80 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:29.322204 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 6c7a48c5a122421797e1924fc36f3c80 (table rebalancer_dynamic_table_new_name [id=2df0556092e945fb9098e7420791ab21]) successfully deleted
I20240503 14:13:29.357985 31316 consensus_queue.cc:1035] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.363210 31502 consensus_queue.cc:1035] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.394766 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:29.402375 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4c447b816b430caf4b213f896e51e3 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.402704 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4c447b816b430caf4b213f896e51e3 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.402937 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4c447b816b430caf4b213f896e51e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.403198 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4c447b816b430caf4b213f896e51e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.403442 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 917de496d0cb4c6d94c6dcf630b90f4a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.403442 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5126a38fc740d49f638d1b3fb8a9c9 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.403854 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 917de496d0cb4c6d94c6dcf630b90f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.403867 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5126a38fc740d49f638d1b3fb8a9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.408021 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 917de496d0cb4c6d94c6dcf630b90f4a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.408459 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 917de496d0cb4c6d94c6dcf630b90f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.410001 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 917de496d0cb4c6d94c6dcf630b90f4a (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.410413 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 917de496d0cb4c6d94c6dcf630b90f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.410770 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 4d4c447b816b430caf4b213f896e51e3 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.411396 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4c447b816b430caf4b213f896e51e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.415797 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5126a38fc740d49f638d1b3fb8a9c9 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.416313 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5126a38fc740d49f638d1b3fb8a9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.419359 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5126a38fc740d49f638d1b3fb8a9c9 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.419914 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5126a38fc740d49f638d1b3fb8a9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.429837 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:29.436914 31508 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:29.438834 31508 tablet_bootstrap.cc:654] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.444819 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 2f00a37fa12a41fc969770f3bb8d5e6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603]), partition=
I20240503 14:13:29.445288 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f00a37fa12a41fc969770f3bb8d5e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.448983 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 2f00a37fa12a41fc969770f3bb8d5e6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603]), partition=
I20240503 14:13:29.449469 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f00a37fa12a41fc969770f3bb8d5e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.452848 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 14:13:29.453022 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 14:13:29.453223 31508 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:29.453397 31508 ts_tablet_manager.cc:1367] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240503 14:13:29.454512 31508 raft_consensus.cc:348] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.454744 31508 raft_consensus.cc:374] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.454836 31508 raft_consensus.cc:729] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.454919 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 2f00a37fa12a41fc969770f3bb8d5e6b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603]), partition=
I20240503 14:13:29.455307 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f00a37fa12a41fc969770f3bb8d5e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.455157 31508 consensus_queue.cc:260] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.456729 31508 ts_tablet_manager.cc:1398] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:29.456933 31509 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:29.456985 31508 tablet_bootstrap.cc:492] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:29.459791 31508 tablet_bootstrap.cc:654] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.463699 31509 tablet_bootstrap.cc:654] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.464372 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 0e4c8fcd60bc42e496d9ca2067a43115
I20240503 14:13:29.465155 28413 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4c8fcd60bc42e496d9ca2067a43115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:29.465476 31306 tablet_replica.cc:330] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:29.465590 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4c8fcd60bc42e496d9ca2067a43115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:29.465709 31306 raft_consensus.cc:2229] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.465979 31306 raft_consensus.cc:2258] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.466008 31310 tablet_replica.cc:330] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:29.466317 31310 raft_consensus.cc:2229] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.466629 31310 raft_consensus.cc:2258] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.466681 31306 ts_tablet_manager.cc:1875] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.467489 31310 ts_tablet_manager.cc:1875] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.467589 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 6bff51b843af4f488cde34745619d3c8
I20240503 14:13:29.468657 28413 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bff51b843af4f488cde34745619d3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:29.468652 28153 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4c8fcd60bc42e496d9ca2067a43115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:29.468876 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bff51b843af4f488cde34745619d3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:29.468950 31311 tablet_replica.cc:330] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:29.469144 31311 raft_consensus.cc:2229] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:29.469183 31488 tablet_replica.cc:330] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:29.469360 31488 raft_consensus.cc:2229] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.469483 31311 raft_consensus.cc:2258] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.469623 31488 raft_consensus.cc:2258] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.470147 31311 ts_tablet_manager.cc:1875] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.470198 31488 ts_tablet_manager.cc:1875] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.470355 31508 tablet_bootstrap.cc:492] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:29.470510 31508 ts_tablet_manager.cc:1367] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20240503 14:13:29.471320 31508 raft_consensus.cc:348] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.472483 31508 raft_consensus.cc:374] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.472573 31508 raft_consensus.cc:729] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.472852 31508 consensus_queue.cc:260] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.475008 31508 ts_tablet_manager.cc:1398] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:29.476078 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bff51b843af4f488cde34745619d3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:29.478426 31311 ts_tablet_manager.cc:1888] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.478562 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 1598708982124ee69f177945753d3703
I20240503 14:13:29.479362 31311 log.cc:1196] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/0e4c8fcd60bc42e496d9ca2067a43115
I20240503 14:13:29.479473 28413 tablet_service.cc:1514] Processing DeleteTablet for tablet 1598708982124ee69f177945753d3703 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:29.479916 31311 ts_tablet_manager.cc:1909] T 0e4c8fcd60bc42e496d9ca2067a43115 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:29.480849 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 0e4c8fcd60bc42e496d9ca2067a43115 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.481133 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 1598708982124ee69f177945753d3703 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:29.482419 28153 tablet_service.cc:1514] Processing DeleteTablet for tablet 1598708982124ee69f177945753d3703 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:29.482671 31311 tablet_replica.cc:330] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:29.482926 31311 raft_consensus.cc:2229] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:29.483793 31311 raft_consensus.cc:2258] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.485395 31311 ts_tablet_manager.cc:1875] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.487231 31310 ts_tablet_manager.cc:1888] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.487475 31310 log.cc:1196] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/0e4c8fcd60bc42e496d9ca2067a43115
I20240503 14:13:29.487965 31310 ts_tablet_manager.cc:1909] T 0e4c8fcd60bc42e496d9ca2067a43115 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:29.489495 31310 tablet_replica.cc:330] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:29.489735 31310 raft_consensus.cc:2229] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.490022 31310 raft_consensus.cc:2258] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.490566 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 0e4c8fcd60bc42e496d9ca2067a43115 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.490969 31310 ts_tablet_manager.cc:1875] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.494169 31509 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:29.495990 31509 ts_tablet_manager.cc:1367] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240503 14:13:29.497817 31488 ts_tablet_manager.cc:1888] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.497960 31488 log.cc:1196] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/6bff51b843af4f488cde34745619d3c8
I20240503 14:13:29.498137 31310 ts_tablet_manager.cc:1888] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.498070 31509 raft_consensus.cc:348] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.498332 31509 raft_consensus.cc:374] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.498389 31310 log.cc:1196] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/6bff51b843af4f488cde34745619d3c8
I20240503 14:13:29.498438 31509 raft_consensus.cc:729] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.498477 31488 ts_tablet_manager.cc:1909] T 6bff51b843af4f488cde34745619d3c8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:29.498975 31310 ts_tablet_manager.cc:1909] T 6bff51b843af4f488cde34745619d3c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:29.499940 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 6bff51b843af4f488cde34745619d3c8 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.500026 31310 tablet_replica.cc:330] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:29.500223 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 6bff51b843af4f488cde34745619d3c8 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.500377 31310 raft_consensus.cc:2229] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.500679 31310 raft_consensus.cc:2258] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.501027 31311 ts_tablet_manager.cc:1888] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.501179 31311 log.cc:1196] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/1598708982124ee69f177945753d3703
I20240503 14:13:29.501233 31306 ts_tablet_manager.cc:1888] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.501366 31306 log.cc:1196] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e4c8fcd60bc42e496d9ca2067a43115
I20240503 14:13:29.501616 31311 ts_tablet_manager.cc:1909] T 1598708982124ee69f177945753d3703 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:29.501806 31306 ts_tablet_manager.cc:1909] T 0e4c8fcd60bc42e496d9ca2067a43115 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:29.501654 31310 ts_tablet_manager.cc:1875] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.502638 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 1598708982124ee69f177945753d3703 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.502727 31306 tablet_replica.cc:330] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:29.502959 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 0e4c8fcd60bc42e496d9ca2067a43115 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.503232 31306 raft_consensus.cc:2229] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:29.503515 31306 raft_consensus.cc:2258] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.503799 31509 consensus_queue.cc:260] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.504212 31306 ts_tablet_manager.cc:1875] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.504416 31509 ts_tablet_manager.cc:1398] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:29.504709 31509 tablet_bootstrap.cc:492] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:29.513429 31514 tablet_bootstrap.cc:492] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:29.514736 31515 tablet_bootstrap.cc:492] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:29.514803 31310 ts_tablet_manager.cc:1888] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.514963 31509 tablet_bootstrap.cc:654] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.515075 31310 log.cc:1196] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/1598708982124ee69f177945753d3703
I20240503 14:13:29.516413 31310 ts_tablet_manager.cc:1909] T 1598708982124ee69f177945753d3703 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:29.516736 31515 tablet_bootstrap.cc:654] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.517467 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 1598708982124ee69f177945753d3703 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.519290 31514 tablet_bootstrap.cc:654] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.522528 31510 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:29.524473 31510 tablet_bootstrap.cc:654] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.525970 31506 tablet_bootstrap.cc:492] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:29.526649 31507 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:29.528088 31507 tablet_bootstrap.cc:654] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.530947 31515 tablet_bootstrap.cc:492] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:29.531162 31515 ts_tablet_manager.cc:1367] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 14:13:29.531847 31515 raft_consensus.cc:348] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.532027 31515 raft_consensus.cc:374] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.532143 31515 raft_consensus.cc:729] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.532348 31515 consensus_queue.cc:260] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.532799 31515 ts_tablet_manager.cc:1398] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:29.534089 31506 tablet_bootstrap.cc:654] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.538216 31514 tablet_bootstrap.cc:492] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:29.538465 31514 ts_tablet_manager.cc:1367] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20240503 14:13:29.539238 31514 raft_consensus.cc:348] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.539513 31514 raft_consensus.cc:374] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.539661 31514 raft_consensus.cc:729] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.540012 31514 consensus_queue.cc:260] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.540774 31514 ts_tablet_manager.cc:1398] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:29.542228 31306 ts_tablet_manager.cc:1888] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.542451 31306 log.cc:1196] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/6bff51b843af4f488cde34745619d3c8
I20240503 14:13:29.542944 31306 ts_tablet_manager.cc:1909] T 6bff51b843af4f488cde34745619d3c8 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:29.543903 31306 tablet_replica.cc:330] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:29.544118 31306 raft_consensus.cc:2229] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.544363 31306 raft_consensus.cc:2258] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.544728 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 6bff51b843af4f488cde34745619d3c8 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.545023 31306 ts_tablet_manager.cc:1875] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.547758 31510 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:29.547911 31510 ts_tablet_manager.cc:1367] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 14:13:29.548594 31510 raft_consensus.cc:348] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.548758 31510 raft_consensus.cc:374] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.549345 31507 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:29.549492 31507 ts_tablet_manager.cc:1367] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240503 14:13:29.550051 31507 raft_consensus.cc:348] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.550227 31507 raft_consensus.cc:374] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.554329 31510 raft_consensus.cc:729] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.546746 31506 tablet_bootstrap.cc:492] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:29.554600 31506 ts_tablet_manager.cc:1367] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240503 14:13:29.554895 31510 consensus_queue.cc:260] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.555912 31506 raft_consensus.cc:348] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.556181 31507 raft_consensus.cc:729] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.556741 31506 raft_consensus.cc:374] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.556933 31506 raft_consensus.cc:729] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.557195 31506 consensus_queue.cc:260] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.559589 31506 ts_tablet_manager.cc:1398] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:29.559865 31506 tablet_bootstrap.cc:492] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:29.561403 31510 ts_tablet_manager.cc:1398] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:29.561434 31506 tablet_bootstrap.cc:654] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.561691 31507 consensus_queue.cc:260] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.563948 31509 tablet_bootstrap.cc:492] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:29.564244 31509 ts_tablet_manager.cc:1367] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.030s
I20240503 14:13:29.565016 31306 ts_tablet_manager.cc:1888] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:29.565155 31306 log.cc:1196] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/1598708982124ee69f177945753d3703
I20240503 14:13:29.565068 31509 raft_consensus.cc:348] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.565515 31509 raft_consensus.cc:374] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.565636 31509 raft_consensus.cc:729] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.565831 31306 ts_tablet_manager.cc:1909] T 1598708982124ee69f177945753d3703 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:29.566002 31509 consensus_queue.cc:260] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.567088 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 1598708982124ee69f177945753d3703 (table rebalance_test_table_10 [id=2ab3e43bfc4c49b1b72a59cb0113ebdd]) successfully deleted
I20240503 14:13:29.567867 31509 ts_tablet_manager.cc:1398] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:29.575294 31510 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:29.577670 31507 ts_tablet_manager.cc:1398] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 14:13:29.578563 31507 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:29.579931 31510 tablet_bootstrap.cc:654] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.581696 31507 tablet_bootstrap.cc:654] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.588533 31506 tablet_bootstrap.cc:492] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:29.588809 31506 ts_tablet_manager.cc:1367] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.015s
I20240503 14:13:29.590221 31506 raft_consensus.cc:348] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.590682 31506 raft_consensus.cc:374] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.590818 31506 raft_consensus.cc:729] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.591229 31506 consensus_queue.cc:260] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.592532 31507 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:29.592459 31506 ts_tablet_manager.cc:1398] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:13:29.594215 31507 ts_tablet_manager.cc:1367] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 14:13:29.595134 31507 raft_consensus.cc:348] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.595427 31507 raft_consensus.cc:374] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.595561 31507 raft_consensus.cc:729] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.595834 31507 consensus_queue.cc:260] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.597180 31507 ts_tablet_manager.cc:1398] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:29.603420 31510 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:29.603575 31510 ts_tablet_manager.cc:1367] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.014s
I20240503 14:13:29.604183 31510 raft_consensus.cc:348] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.604369 31510 raft_consensus.cc:374] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.604454 31510 raft_consensus.cc:729] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.604720 31510 consensus_queue.cc:260] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.605159 31510 ts_tablet_manager.cc:1398] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:29.633361 31505 raft_consensus.cc:482] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.633527 31505 raft_consensus.cc:504] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.634627 31505 leader_election.cc:288] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:29.643668 31392 raft_consensus.cc:482] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.643842 31392 raft_consensus.cc:504] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.644541 31392 leader_election.cc:288] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:29.645754 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f00a37fa12a41fc969770f3bb8d5e6b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:29.645751 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f00a37fa12a41fc969770f3bb8d5e6b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:29.645988 28432 raft_consensus.cc:2454] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:29.646593 28042 raft_consensus.cc:2454] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:29.646977 28110 leader_election.cc:302] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:29.649072 31392 raft_consensus.cc:2781] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.649227 31392 raft_consensus.cc:482] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.649330 31392 raft_consensus.cc:3037] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.652933 31392 raft_consensus.cc:504] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.654103 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f00a37fa12a41fc969770f3bb8d5e6b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:29.654389 28432 raft_consensus.cc:3037] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.657454 28432 raft_consensus.cc:2454] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:29.657995 28110 leader_election.cc:302] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:29.658352 31452 raft_consensus.cc:2781] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.661908 31392 leader_election.cc:288] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:29.661865 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f00a37fa12a41fc969770f3bb8d5e6b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:29.662088 28042 raft_consensus.cc:3037] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.666796 28042 raft_consensus.cc:2454] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:29.670607 31452 raft_consensus.cc:686] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:29.671036 31452 consensus_queue.cc:237] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:29.680441 31413 raft_consensus.cc:482] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.680649 31413 raft_consensus.cc:504] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.681289 31413 leader_election.cc:288] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:29.681715 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:29.681980 31202 raft_consensus.cc:2454] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:29.682406 30875 leader_election.cc:302] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:29.682704 31413 raft_consensus.cc:2781] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.682813 31413 raft_consensus.cc:482] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.682884 31413 raft_consensus.cc:3037] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.683481 27899 catalog_manager.cc:5484] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:29.685748 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:29.686801 31413 raft_consensus.cc:504] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.687719 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:29.688793 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:29.689028 31202 raft_consensus.cc:3037] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.685989 28432 raft_consensus.cc:2454] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:29.691982 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:29.692090 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:29.692257 30941 raft_consensus.cc:2379] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5388902973134958a030fd63ab4faba4 in current term 1: Already voted for candidate 6bbc686598f24f518d7ca6bb2063058d in this term.
I20240503 14:13:29.691865 31202 raft_consensus.cc:2454] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:29.692934 28368 leader_election.cc:302] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:13:29.693110 30875 leader_election.cc:302] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters: 5388902973134958a030fd63ab4faba4
I20240503 14:13:29.693337 31505 raft_consensus.cc:3037] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.693409 31339 raft_consensus.cc:2781] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.693544 31339 raft_consensus.cc:686] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:29.693822 31339 consensus_queue.cc:237] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.696489 31505 raft_consensus.cc:2726] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:29.696756 31413 leader_election.cc:288] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:29.699167 27899 catalog_manager.cc:5484] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:29.714409 31346 raft_consensus.cc:482] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.714591 31346 raft_consensus.cc:504] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.715162 31346 leader_election.cc:288] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:29.715626 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917de496d0cb4c6d94c6dcf630b90f4a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:29.715682 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917de496d0cb4c6d94c6dcf630b90f4a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:29.715842 31202 raft_consensus.cc:2454] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:29.715893 30810 raft_consensus.cc:2454] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:29.716277 31006 leader_election.cc:302] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:29.716518 31346 raft_consensus.cc:2781] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.716603 31346 raft_consensus.cc:482] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.716722 31346 raft_consensus.cc:3037] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.718511 31346 raft_consensus.cc:504] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.719015 31346 leader_election.cc:288] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:29.719467 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917de496d0cb4c6d94c6dcf630b90f4a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:29.719622 31202 raft_consensus.cc:3037] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.719517 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "917de496d0cb4c6d94c6dcf630b90f4a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:29.719808 30810 raft_consensus.cc:3037] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.721434 31202 raft_consensus.cc:2454] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:29.721737 31006 leader_election.cc:302] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:29.721940 31346 raft_consensus.cc:2781] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.722348 31346 raft_consensus.cc:686] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:29.722688 31346 consensus_queue.cc:237] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:29.723788 30810 raft_consensus.cc:2454] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:29.726266 27899 catalog_manager.cc:5484] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:29.752430 31473 raft_consensus.cc:482] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.752650 31473 raft_consensus.cc:504] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.753365 31473 leader_election.cc:288] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:29.753754 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4c447b816b430caf4b213f896e51e3" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:29.753875 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4c447b816b430caf4b213f896e51e3" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:29.754040 31071 raft_consensus.cc:2454] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:29.754127 30941 raft_consensus.cc:2454] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:29.754503 30745 leader_election.cc:302] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:29.754833 31473 raft_consensus.cc:2781] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.754961 31473 raft_consensus.cc:482] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.755095 31473 raft_consensus.cc:3037] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.758575 31473 raft_consensus.cc:504] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.759459 31473 leader_election.cc:288] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:29.759655 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4c447b816b430caf4b213f896e51e3" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:29.759872 31071 raft_consensus.cc:3037] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.759945 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d4c447b816b430caf4b213f896e51e3" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:29.760212 30941 raft_consensus.cc:3037] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.761822 31071 raft_consensus.cc:2454] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:29.762228 30745 leader_election.cc:302] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:29.762487 31473 raft_consensus.cc:2781] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.762904 31473 raft_consensus.cc:686] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:29.763222 31473 consensus_queue.cc:237] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:29.763625 30941 raft_consensus.cc:2454] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:29.766041 27899 catalog_manager.cc:5484] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:29.814649 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:29.832892 28434 raft_consensus.cc:1261] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:29.833571 28042 raft_consensus.cc:1261] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:29.835605 31392 consensus_queue.cc:1035] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.836073 31392 consensus_queue.cc:1035] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.844854 27894 catalog_manager.cc:5576] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293) does not have the latest schema for tablet 2f00a37fa12a41fc969770f3bb8d5e6b (table rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603]). Expected version 1 got 0
I20240503 14:13:29.844844 31540 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:29.845494 31540 tablet.cc:1620] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:29.855569 31539 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:29.858152 31539 tablet.cc:1620] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:29.865283 27894 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 2f00a37fa12a41fc969770f3bb8d5e6b (table rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603]). Expected version 1 got 0
I20240503 14:13:29.872877 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603] alter complete (version 1)
I20240503 14:13:29.879406 27899 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 2f00a37fa12a41fc969770f3bb8d5e6b (table rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603]). Expected version 1 got 0
I20240503 14:13:29.876412 31542 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:29.881809 31542 tablet.cc:1620] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:29.976511 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:29.982751 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 14:13:29.982954 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 14:13:29.984864 31539 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.985352 31539 tablet.cc:1620] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:29.987303 31542 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.987718 31542 tablet.cc:1620] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:29.988561 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 6f4bd9f587564718a017f0ab12d645d5
I20240503 14:13:29.988847 31540 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.989344 31540 tablet.cc:1620] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:29.989815 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4bd9f587564718a017f0ab12d645d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:29.989917 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4bd9f587564718a017f0ab12d645d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:29.990172 31311 tablet_replica.cc:330] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:29.990190 31310 tablet_replica.cc:330] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:29.990383 31311 raft_consensus.cc:2229] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.990382 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f4bd9f587564718a017f0ab12d645d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:29.990615 31311 raft_consensus.cc:2258] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.990648 31306 tablet_replica.cc:330] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:29.990921 31306 raft_consensus.cc:2229] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.991007 31310 raft_consensus.cc:2229] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:29.991328 31311 ts_tablet_manager.cc:1875] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.991536 31306 raft_consensus.cc:2258] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.993856 31306 ts_tablet_manager.cc:1875] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.994580 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 7a4eb5689908428398f53dc484ea8072
I20240503 14:13:29.995287 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet ee57689f4f764553bbfc9838e19de76d
I20240503 14:13:29.997476 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet ee57689f4f764553bbfc9838e19de76d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:29.998055 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4eb5689908428398f53dc484ea8072 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:29.998925 31310 raft_consensus.cc:2258] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.999522 31310 ts_tablet_manager.cc:1875] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.000322 31311 ts_tablet_manager.cc:1888] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.000432 31311 log.cc:1196] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/6f4bd9f587564718a017f0ab12d645d5
I20240503 14:13:30.000912 31311 ts_tablet_manager.cc:1909] T 6f4bd9f587564718a017f0ab12d645d5 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:30.001713 31311 tablet_replica.cc:330] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:30.001818 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 6f4bd9f587564718a017f0ab12d645d5 (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.001942 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=c3e8ceb0120140b5bba4d7aa26524603] alter complete (version 2)
I20240503 14:13:30.002790 31311 raft_consensus.cc:2229] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.002990 31311 raft_consensus.cc:2258] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.003423 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4eb5689908428398f53dc484ea8072 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:30.003522 31311 ts_tablet_manager.cc:1875] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.003590 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet ee57689f4f764553bbfc9838e19de76d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:30.004158 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4eb5689908428398f53dc484ea8072 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:30.005244 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet ee57689f4f764553bbfc9838e19de76d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:30.009374 31306 ts_tablet_manager.cc:1888] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.009575 31306 log.cc:1196] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f4bd9f587564718a017f0ab12d645d5
I20240503 14:13:30.010497 31311 ts_tablet_manager.cc:1888] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.010614 31543 tablet_replica.cc:330] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:30.010637 31311 log.cc:1196] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a4eb5689908428398f53dc484ea8072
I20240503 14:13:30.010876 31543 raft_consensus.cc:2229] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:30.011025 31311 ts_tablet_manager.cc:1909] T 7a4eb5689908428398f53dc484ea8072 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:30.011267 31543 raft_consensus.cc:2258] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.011410 31310 ts_tablet_manager.cc:1888] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.011550 31310 log.cc:1196] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f4bd9f587564718a017f0ab12d645d5
I20240503 14:13:30.011940 31310 ts_tablet_manager.cc:1909] T 6f4bd9f587564718a017f0ab12d645d5 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:30.011966 31543 ts_tablet_manager.cc:1875] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.012087 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 7a4eb5689908428398f53dc484ea8072 (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.012723 31310 tablet_replica.cc:330] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:30.012810 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 6f4bd9f587564718a017f0ab12d645d5 (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.012920 31310 raft_consensus.cc:2229] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.013130 31310 raft_consensus.cc:2258] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.013617 31310 ts_tablet_manager.cc:1875] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.016312 31543 ts_tablet_manager.cc:1888] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.016427 31543 log.cc:1196] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a4eb5689908428398f53dc484ea8072
I20240503 14:13:30.016763 31543 ts_tablet_manager.cc:1909] T 7a4eb5689908428398f53dc484ea8072 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:30.017421 31543 tablet_replica.cc:330] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:30.017565 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 7a4eb5689908428398f53dc484ea8072 (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.017604 31310 ts_tablet_manager.cc:1888] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.017649 31543 raft_consensus.cc:2229] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:30.017694 31310 log.cc:1196] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/7a4eb5689908428398f53dc484ea8072
I20240503 14:13:30.017899 31543 raft_consensus.cc:2258] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.018020 31310 ts_tablet_manager.cc:1909] T 7a4eb5689908428398f53dc484ea8072 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:30.018481 31543 ts_tablet_manager.cc:1875] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.018730 31310 tablet_replica.cc:330] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:30.018900 31306 ts_tablet_manager.cc:1909] T 6f4bd9f587564718a017f0ab12d645d5 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:30.018924 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 7a4eb5689908428398f53dc484ea8072 (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.019670 31306 tablet_replica.cc:330] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:30.019802 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 6f4bd9f587564718a017f0ab12d645d5 (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.019966 31306 raft_consensus.cc:2229] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.020216 31306 raft_consensus.cc:2258] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.020767 31306 ts_tablet_manager.cc:1875] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.022626 31543 ts_tablet_manager.cc:1888] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.022763 31543 log.cc:1196] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee57689f4f764553bbfc9838e19de76d
I20240503 14:13:30.023083 31543 ts_tablet_manager.cc:1909] T ee57689f4f764553bbfc9838e19de76d P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:30.023975 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet ee57689f4f764553bbfc9838e19de76d (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.024967 31306 ts_tablet_manager.cc:1888] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.025074 31306 log.cc:1196] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee57689f4f764553bbfc9838e19de76d
I20240503 14:13:30.025259 31310 raft_consensus.cc:2229] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.025429 31306 ts_tablet_manager.cc:1909] T ee57689f4f764553bbfc9838e19de76d P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:30.025480 31310 raft_consensus.cc:2258] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.026036 31310 ts_tablet_manager.cc:1875] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.026216 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet ee57689f4f764553bbfc9838e19de76d (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.030062 31310 ts_tablet_manager.cc:1888] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.030205 31310 log.cc:1196] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee57689f4f764553bbfc9838e19de76d
I20240503 14:13:30.030663 31310 ts_tablet_manager.cc:1909] T ee57689f4f764553bbfc9838e19de76d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:30.031450 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet ee57689f4f764553bbfc9838e19de76d (table rebalance_test_table_12 [id=54331f97a1ea4cbc912d27090bd65040]) successfully deleted
I20240503 14:13:30.073077 31339 consensus_queue.cc:1035] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.078379 31339 consensus_queue.cc:1035] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.106828 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:30.113050 31539 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:30.113786 31540 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:30.115164 31542 tablet.cc:1722] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:30.122635 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=c3e8ceb0120140b5bba4d7aa26524603] alter complete (version 3)
I20240503 14:13:30.211246 31346 consensus_queue.cc:1035] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.215337 31346 consensus_queue.cc:1035] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.225306 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:30.225535 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:30.228159 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 2f00a37fa12a41fc969770f3bb8d5e6b
I20240503 14:13:30.228866 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f00a37fa12a41fc969770f3bb8d5e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:30.229118 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f00a37fa12a41fc969770f3bb8d5e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:30.229182 31311 tablet_replica.cc:330] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:30.229403 31306 tablet_replica.cc:330] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:30.229408 31311 raft_consensus.cc:2229] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:30.229458 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f00a37fa12a41fc969770f3bb8d5e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:30.229631 31306 raft_consensus.cc:2229] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.229748 31310 tablet_replica.cc:330] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:30.229843 31306 raft_consensus.cc:2258] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.229859 31311 raft_consensus.cc:2258] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.229954 31310 raft_consensus.cc:2229] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.230175 31310 raft_consensus.cc:2258] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.230705 31311 ts_tablet_manager.cc:1875] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.230839 31306 ts_tablet_manager.cc:1875] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.231001 31310 ts_tablet_manager.cc:1875] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.236665 31311 ts_tablet_manager.cc:1888] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:30.236799 31311 log.cc:1196] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f00a37fa12a41fc969770f3bb8d5e6b
I20240503 14:13:30.237082 31306 ts_tablet_manager.cc:1888] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:30.237179 31311 ts_tablet_manager.cc:1909] T 2f00a37fa12a41fc969770f3bb8d5e6b P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:30.237219 31306 log.cc:1196] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/2f00a37fa12a41fc969770f3bb8d5e6b
I20240503 14:13:30.237674 31306 ts_tablet_manager.cc:1909] T 2f00a37fa12a41fc969770f3bb8d5e6b P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:30.237671 31310 ts_tablet_manager.cc:1888] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:30.237782 31310 log.cc:1196] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f00a37fa12a41fc969770f3bb8d5e6b
I20240503 14:13:30.238277 31310 ts_tablet_manager.cc:1909] T 2f00a37fa12a41fc969770f3bb8d5e6b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:30.239010 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 2f00a37fa12a41fc969770f3bb8d5e6b (table rebalancer_dynamic_table_new_name [id=c3e8ceb0120140b5bba4d7aa26524603]) successfully deleted
I20240503 14:13:30.239018 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 2f00a37fa12a41fc969770f3bb8d5e6b (table rebalancer_dynamic_table_new_name [id=c3e8ceb0120140b5bba4d7aa26524603]) successfully deleted
I20240503 14:13:30.239331 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 2f00a37fa12a41fc969770f3bb8d5e6b (table rebalancer_dynamic_table_new_name [id=c3e8ceb0120140b5bba4d7aa26524603]) successfully deleted
I20240503 14:13:30.298604 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:30.306180 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 84dbd48ddf494ae2b9e23ba2b294376c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.306532 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 84dbd48ddf494ae2b9e23ba2b294376c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.306689 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84dbd48ddf494ae2b9e23ba2b294376c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.306944 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84dbd48ddf494ae2b9e23ba2b294376c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.307854 28154 tablet_service.cc:1467] Processing CreateTablet for tablet a82470f287274b1c878089fa9d56ef40 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.308281 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82470f287274b1c878089fa9d56ef40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.310964 30921 tablet_service.cc:1467] Processing CreateTablet for tablet a82470f287274b1c878089fa9d56ef40 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.311348 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82470f287274b1c878089fa9d56ef40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.311234 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 84dbd48ddf494ae2b9e23ba2b294376c (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.311249 30789 tablet_service.cc:1467] Processing CreateTablet for tablet a82470f287274b1c878089fa9d56ef40 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.311601 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82470f287274b1c878089fa9d56ef40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.311618 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84dbd48ddf494ae2b9e23ba2b294376c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.314545 30920 tablet_service.cc:1467] Processing CreateTablet for tablet a3888675106d4d389718380627c95edf (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.314914 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3888675106d4d389718380627c95edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.314769 31051 tablet_service.cc:1467] Processing CreateTablet for tablet a3888675106d4d389718380627c95edf (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.315301 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3888675106d4d389718380627c95edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.319101 28412 tablet_service.cc:1467] Processing CreateTablet for tablet a3888675106d4d389718380627c95edf (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:30.319476 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3888675106d4d389718380627c95edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.320683 31555 tablet_bootstrap.cc:492] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:30.322352 31555 tablet_bootstrap.cc:654] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.328635 31559 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:30.329039 31557 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:30.331842 31559 tablet_bootstrap.cc:654] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.332469 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:30.335649 31555 tablet_bootstrap.cc:492] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:30.335803 31555 ts_tablet_manager.cc:1367] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20240503 14:13:30.336566 31555 raft_consensus.cc:348] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.336779 31555 raft_consensus.cc:374] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.336879 31555 raft_consensus.cc:729] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.337101 31555 consensus_queue.cc:260] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.337652 31555 ts_tablet_manager.cc:1398] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:30.338721 31554 tablet_bootstrap.cc:492] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:30.341418 31554 tablet_bootstrap.cc:654] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.344249 30788 tablet_service.cc:1467] Processing CreateTablet for tablet 0f591486bd3d4b60a7d2f97bac619845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b]), partition=
I20240503 14:13:30.344719 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f591486bd3d4b60a7d2f97bac619845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.346640 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 0f591486bd3d4b60a7d2f97bac619845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b]), partition=
I20240503 14:13:30.347040 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f591486bd3d4b60a7d2f97bac619845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.347755 31559 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:30.347891 31559 ts_tablet_manager.cc:1367] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20240503 14:13:30.348649 31559 raft_consensus.cc:348] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.349181 31559 raft_consensus.cc:374] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.349270 31559 raft_consensus.cc:729] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.349486 31559 consensus_queue.cc:260] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.350157 31559 ts_tablet_manager.cc:1398] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:30.360997 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 0f591486bd3d4b60a7d2f97bac619845 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b]), partition=
I20240503 14:13:30.361267 31561 tablet_bootstrap.cc:492] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:30.361425 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f591486bd3d4b60a7d2f97bac619845. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.362387 31473 consensus_queue.cc:1035] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.363406 31561 tablet_bootstrap.cc:654] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.365927 31557 tablet_bootstrap.cc:654] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.366436 31473 consensus_queue.cc:1035] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.378163 31561 tablet_bootstrap.cc:492] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:30.378521 31561 ts_tablet_manager.cc:1367] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.006s
I20240503 14:13:30.378898 31557 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:30.379091 31557 ts_tablet_manager.cc:1367] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.002s
I20240503 14:13:30.379649 31561 raft_consensus.cc:348] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.379966 31561 raft_consensus.cc:374] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.380108 31561 raft_consensus.cc:729] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.380280 31557 raft_consensus.cc:348] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.380595 31557 raft_consensus.cc:374] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.380438 31561 consensus_queue.cc:260] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.380743 31557 raft_consensus.cc:729] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.382191 31561 ts_tablet_manager.cc:1398] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:30.381192 31557 consensus_queue.cc:260] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.390957 31557 ts_tablet_manager.cc:1398] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.009s user 0.001s sys 0.003s
I20240503 14:13:30.394438 31557 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:30.399876 31557 tablet_bootstrap.cc:654] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.406697 31554 tablet_bootstrap.cc:492] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:30.406973 31554 ts_tablet_manager.cc:1367] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20240503 14:13:30.408214 31554 raft_consensus.cc:348] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.408602 31554 raft_consensus.cc:374] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.408753 31554 raft_consensus.cc:729] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.409673 31558 tablet_bootstrap.cc:492] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:30.410388 31554 consensus_queue.cc:260] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.411583 31554 ts_tablet_manager.cc:1398] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:30.413312 31558 tablet_bootstrap.cc:654] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.423110 31558 tablet_bootstrap.cc:492] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:30.424814 31558 ts_tablet_manager.cc:1367] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 14:13:30.426014 31554 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:30.426064 31558 raft_consensus.cc:348] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.428347 31558 raft_consensus.cc:374] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.428761 31558 raft_consensus.cc:729] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.429616 31558 consensus_queue.cc:260] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.432425 31567 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:30.434651 31554 tablet_bootstrap.cc:654] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.434677 31558 ts_tablet_manager.cc:1398] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20240503 14:13:30.436004 31558 tablet_bootstrap.cc:492] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:30.436555 31567 tablet_bootstrap.cc:654] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.438763 31558 tablet_bootstrap.cc:654] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.442837 31554 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:30.443065 31554 ts_tablet_manager.cc:1367] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20240503 14:13:30.445864 31557 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:30.446089 31557 ts_tablet_manager.cc:1367] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.005s
I20240503 14:13:30.446974 31557 raft_consensus.cc:348] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.447305 31557 raft_consensus.cc:374] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.447434 31557 raft_consensus.cc:729] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.449162 31557 consensus_queue.cc:260] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.449146 31554 raft_consensus.cc:348] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.449431 31554 raft_consensus.cc:374] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.449517 31554 raft_consensus.cc:729] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.449771 31557 ts_tablet_manager.cc:1398] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:30.449765 31554 consensus_queue.cc:260] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.450415 31554 ts_tablet_manager.cc:1398] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:13:30.451715 31558 tablet_bootstrap.cc:492] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:30.451939 31558 ts_tablet_manager.cc:1367] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20240503 14:13:30.452749 31558 raft_consensus.cc:348] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.453053 31558 raft_consensus.cc:374] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.453171 31558 raft_consensus.cc:729] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.453480 31558 consensus_queue.cc:260] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.454609 31558 ts_tablet_manager.cc:1398] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:30.454952 31558 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:30.456605 31558 tablet_bootstrap.cc:654] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.457584 31567 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:30.457741 31567 ts_tablet_manager.cc:1367] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.004s
I20240503 14:13:30.458586 31567 raft_consensus.cc:348] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.458847 31567 raft_consensus.cc:374] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.458950 31567 raft_consensus.cc:729] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.459177 31567 consensus_queue.cc:260] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.459638 31567 ts_tablet_manager.cc:1398] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:30.460206 31567 tablet_bootstrap.cc:492] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:30.462270 31567 tablet_bootstrap.cc:654] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.464303 31567 tablet_bootstrap.cc:492] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:30.464473 31567 ts_tablet_manager.cc:1367] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:13:30.465341 31567 raft_consensus.cc:348] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.465903 31567 raft_consensus.cc:374] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.465978 31567 raft_consensus.cc:729] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.466183 31567 consensus_queue.cc:260] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.466754 31567 ts_tablet_manager.cc:1398] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:30.468544 31558 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:30.469511 31558 ts_tablet_manager.cc:1367] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240503 14:13:30.470307 31558 raft_consensus.cc:348] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.470471 31558 raft_consensus.cc:374] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.470566 31558 raft_consensus.cc:729] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.470774 31558 consensus_queue.cc:260] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.471228 31558 ts_tablet_manager.cc:1398] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:30.498399 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 14:13:30.498685 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 14:13:30.501842 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet e8dbe7ee1e7b4dd792a04c87225cb4e8
I20240503 14:13:30.502400 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 8307390c433d4da8bf395b3f6613ca8a
I20240503 14:13:30.502547 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:30.502668 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:30.502784 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 2055aec2f7284c3c8cb772eca6c1513d
I20240503 14:13:30.502805 31310 tablet_replica.cc:330] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:30.502923 31311 tablet_replica.cc:330] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:30.503027 31310 raft_consensus.cc:2229] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.503137 31311 raft_consensus.cc:2229] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:30.503268 31310 raft_consensus.cc:2258] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.503463 31311 raft_consensus.cc:2258] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.503743 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8307390c433d4da8bf395b3f6613ca8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:30.503831 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 2055aec2f7284c3c8cb772eca6c1513d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:30.504069 31310 ts_tablet_manager.cc:1875] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.504124 31311 ts_tablet_manager.cc:1875] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.508278 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:30.508584 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet 8307390c433d4da8bf395b3f6613ca8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:30.508610 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 2055aec2f7284c3c8cb772eca6c1513d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:30.509282 31543 tablet_replica.cc:330] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:30.509460 31543 raft_consensus.cc:2229] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.509687 31543 raft_consensus.cc:2258] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.510283 31543 ts_tablet_manager.cc:1875] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.510973 31310 ts_tablet_manager.cc:1888] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.511185 31310 log.cc:1196] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/e8dbe7ee1e7b4dd792a04c87225cb4e8
I20240503 14:13:30.511649 31310 ts_tablet_manager.cc:1909] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:30.512638 31310 tablet_replica.cc:330] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:30.512694 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.512846 31310 raft_consensus.cc:2229] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.513077 31310 raft_consensus.cc:2258] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.513685 31310 ts_tablet_manager.cc:1875] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.514752 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 2055aec2f7284c3c8cb772eca6c1513d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:30.515151 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 8307390c433d4da8bf395b3f6613ca8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:30.515480 31306 tablet_replica.cc:330] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:30.515672 31306 raft_consensus.cc:2229] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:30.516017 31306 raft_consensus.cc:2258] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.516681 31306 ts_tablet_manager.cc:1875] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.519595 31310 ts_tablet_manager.cc:1888] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.519781 31310 log.cc:1196] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/8307390c433d4da8bf395b3f6613ca8a
I20240503 14:13:30.520236 31310 ts_tablet_manager.cc:1909] T 8307390c433d4da8bf395b3f6613ca8a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:30.520514 31311 ts_tablet_manager.cc:1888] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.521004 31543 ts_tablet_manager.cc:1888] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.521049 31310 tablet_replica.cc:330] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:30.521117 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 8307390c433d4da8bf395b3f6613ca8a (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.521257 31310 raft_consensus.cc:2229] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.521273 31311 log.cc:1196] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8dbe7ee1e7b4dd792a04c87225cb4e8
I20240503 14:13:30.521507 31310 raft_consensus.cc:2258] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.521759 31311 ts_tablet_manager.cc:1909] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:30.521912 31543 log.cc:1196] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/e8dbe7ee1e7b4dd792a04c87225cb4e8
I20240503 14:13:30.522177 31310 ts_tablet_manager.cc:1875] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.523362 31543 ts_tablet_manager.cc:1909] T e8dbe7ee1e7b4dd792a04c87225cb4e8 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:30.524433 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.524916 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet e8dbe7ee1e7b4dd792a04c87225cb4e8 (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.525242 31311 tablet_replica.cc:330] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:30.525501 31311 raft_consensus.cc:2229] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:30.525734 31543 tablet_replica.cc:330] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:30.526094 31311 raft_consensus.cc:2258] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.526394 31543 raft_consensus.cc:2229] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.526405 31306 ts_tablet_manager.cc:1888] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.526970 31306 log.cc:1196] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/8307390c433d4da8bf395b3f6613ca8a
I20240503 14:13:30.527051 31311 ts_tablet_manager.cc:1875] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.527324 31543 raft_consensus.cc:2258] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.527453 31306 ts_tablet_manager.cc:1909] T 8307390c433d4da8bf395b3f6613ca8a P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:30.527876 31543 ts_tablet_manager.cc:1875] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.528380 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 8307390c433d4da8bf395b3f6613ca8a (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.528721 31310 ts_tablet_manager.cc:1888] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.528882 31310 log.cc:1196] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/2055aec2f7284c3c8cb772eca6c1513d
I20240503 14:13:30.529307 31310 ts_tablet_manager.cc:1909] T 2055aec2f7284c3c8cb772eca6c1513d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:30.530172 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 2055aec2f7284c3c8cb772eca6c1513d (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.532990 31543 ts_tablet_manager.cc:1888] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.533116 31543 log.cc:1196] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/8307390c433d4da8bf395b3f6613ca8a
I20240503 14:13:30.533200 31311 ts_tablet_manager.cc:1888] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.533311 31311 log.cc:1196] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/2055aec2f7284c3c8cb772eca6c1513d
I20240503 14:13:30.533501 31543 ts_tablet_manager.cc:1909] T 8307390c433d4da8bf395b3f6613ca8a P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:30.533691 31311 ts_tablet_manager.cc:1909] T 2055aec2f7284c3c8cb772eca6c1513d P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:30.534272 31543 tablet_replica.cc:330] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:30.534315 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 8307390c433d4da8bf395b3f6613ca8a (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.534471 31543 raft_consensus.cc:2229] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.534627 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 2055aec2f7284c3c8cb772eca6c1513d (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.534690 31543 raft_consensus.cc:2258] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.535180 31543 ts_tablet_manager.cc:1875] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.539223 31543 ts_tablet_manager.cc:1888] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:30.539332 31543 log.cc:1196] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/2055aec2f7284c3c8cb772eca6c1513d
I20240503 14:13:30.539651 31543 ts_tablet_manager.cc:1909] T 2055aec2f7284c3c8cb772eca6c1513d P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:30.540396 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 2055aec2f7284c3c8cb772eca6c1513d (table rebalance_test_table_14 [id=831c9776d3e040f1935707cb874a809c]) successfully deleted
I20240503 14:13:30.619874 31549 raft_consensus.cc:482] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.620064 31549 raft_consensus.cc:504] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.620852 31549 leader_election.cc:288] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:30.621318 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dbd48ddf494ae2b9e23ba2b294376c" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:30.621568 28040 raft_consensus.cc:2454] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:30.622040 28370 leader_election.cc:302] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:30.622423 31549 raft_consensus.cc:2781] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.622547 31549 raft_consensus.cc:482] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.622654 31549 raft_consensus.cc:3037] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.625234 31549 raft_consensus.cc:504] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.626024 31549 leader_election.cc:288] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:30.626287 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dbd48ddf494ae2b9e23ba2b294376c" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:30.626490 28040 raft_consensus.cc:3037] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.628697 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dbd48ddf494ae2b9e23ba2b294376c" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:30.628914 30810 raft_consensus.cc:2454] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:30.628886 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dbd48ddf494ae2b9e23ba2b294376c" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:30.629052 28040 raft_consensus.cc:2454] T 84dbd48ddf494ae2b9e23ba2b294376c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:30.629123 30809 raft_consensus.cc:3037] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.629426 28370 leader_election.cc:302] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:30.629943 31549 raft_consensus.cc:2781] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.630065 31549 raft_consensus.cc:686] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:30.630362 31549 consensus_queue.cc:237] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.631978 30809 raft_consensus.cc:2454] T 84dbd48ddf494ae2b9e23ba2b294376c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:30.632200 31544 raft_consensus.cc:482] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.632361 31544 raft_consensus.cc:504] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.633694 27896 catalog_manager.cc:5484] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:30.634187 31544 leader_election.cc:288] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:30.656137 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:30.656374 30809 raft_consensus.cc:2454] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:30.656810 31136 leader_election.cc:302] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:30.657158 31544 raft_consensus.cc:2781] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.657275 31544 raft_consensus.cc:482] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.657367 31544 raft_consensus.cc:3037] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.659082 31346 raft_consensus.cc:482] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.659255 31346 raft_consensus.cc:504] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.659855 31346 leader_election.cc:288] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:30.660123 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:30.660348 30809 raft_consensus.cc:2454] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:30.660456 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:30.660737 31005 leader_election.cc:302] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:30.660974 31346 raft_consensus.cc:2781] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.661096 31346 raft_consensus.cc:482] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.661190 31346 raft_consensus.cc:3037] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.661330 31544 raft_consensus.cc:504] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.661751 31202 raft_consensus.cc:2379] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fde134d0fa2b4900b9badd2e08fea09d in current term 1: Already voted for candidate 530893a961af4b95b6b687fa5ed22f9d in this term.
I20240503 14:13:30.661984 31544 leader_election.cc:288] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:30.667585 31346 raft_consensus.cc:504] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.669512 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:30.669576 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:30.669715 30809 raft_consensus.cc:3037] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.669827 31202 raft_consensus.cc:2379] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fde134d0fa2b4900b9badd2e08fea09d in current term 1: Already voted for candidate 530893a961af4b95b6b687fa5ed22f9d in this term.
I20240503 14:13:30.670548 31346 leader_election.cc:288] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:30.671459 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:30.672508 30809 raft_consensus.cc:2454] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:30.673002 31005 leader_election.cc:302] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters: 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:13:30.673313 31346 raft_consensus.cc:2781] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.673478 31346 raft_consensus.cc:686] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:30.673771 31346 consensus_queue.cc:237] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:30.674024 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:30.674151 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f591486bd3d4b60a7d2f97bac619845" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:30.675136 31138 leader_election.cc:302] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:30.675424 31544 raft_consensus.cc:2726] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:13:30.676805 27896 catalog_manager.cc:5484] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:30.732056 31392 raft_consensus.cc:482] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.732218 31392 raft_consensus.cc:504] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.732945 31392 leader_election.cc:288] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:30.739614 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82470f287274b1c878089fa9d56ef40" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:30.739881 30941 raft_consensus.cc:2454] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:30.740309 28109 leader_election.cc:302] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:30.740289 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82470f287274b1c878089fa9d56ef40" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:30.740538 30809 raft_consensus.cc:2454] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:30.740618 31392 raft_consensus.cc:2781] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.740736 31392 raft_consensus.cc:482] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.740816 31392 raft_consensus.cc:3037] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.742828 31392 raft_consensus.cc:504] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.743322 31392 leader_election.cc:288] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:30.743831 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82470f287274b1c878089fa9d56ef40" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:30.743815 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82470f287274b1c878089fa9d56ef40" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:30.744016 30941 raft_consensus.cc:3037] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.744045 30809 raft_consensus.cc:3037] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.746832 30809 raft_consensus.cc:2454] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:30.746850 30941 raft_consensus.cc:2454] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:30.747210 28107 leader_election.cc:302] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:30.747453 31392 raft_consensus.cc:2781] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.747823 31392 raft_consensus.cc:686] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:30.748180 31392 consensus_queue.cc:237] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:30.750957 27896 catalog_manager.cc:5484] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:30.778738 31346 raft_consensus.cc:482] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.778939 31346 raft_consensus.cc:504] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.779749 31346 leader_election.cc:288] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:30.787366 31339 raft_consensus.cc:482] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.787382 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:30.787569 31339 raft_consensus.cc:504] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.787945 30941 raft_consensus.cc:2454] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:30.788304 31339 leader_election.cc:288] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:30.788384 31007 leader_election.cc:302] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:30.788759 31346 raft_consensus.cc:2781] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.788875 31346 raft_consensus.cc:482] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.788983 31346 raft_consensus.cc:3037] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.788949 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:30.789191 28434 raft_consensus.cc:2454] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:30.789359 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:30.789602 28432 raft_consensus.cc:2454] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:30.789644 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:30.790169 30877 leader_election.cc:302] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:30.790474 31339 raft_consensus.cc:2781] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.790568 31339 raft_consensus.cc:482] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.790666 31339 raft_consensus.cc:3037] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.791519 31346 raft_consensus.cc:504] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.791940 31072 raft_consensus.cc:2379] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bbc686598f24f518d7ca6bb2063058d in current term 1: Already voted for candidate fde134d0fa2b4900b9badd2e08fea09d in this term.
I20240503 14:13:30.792223 31346 leader_election.cc:288] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:30.792551 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:30.792788 31339 raft_consensus.cc:504] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.793169 30941 raft_consensus.cc:2379] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fde134d0fa2b4900b9badd2e08fea09d in current term 1: Already voted for candidate 6bbc686598f24f518d7ca6bb2063058d in this term.
I20240503 14:13:30.793193 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:30.793522 28432 raft_consensus.cc:3037] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.793563 31339 leader_election.cc:288] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:30.793823 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:30.794034 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3888675106d4d389718380627c95edf" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:30.794371 31072 raft_consensus.cc:2379] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bbc686598f24f518d7ca6bb2063058d in current term 1: Already voted for candidate fde134d0fa2b4900b9badd2e08fea09d in this term.
I20240503 14:13:30.794739 30876 leader_election.cc:302] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d; no voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:30.795064 31339 raft_consensus.cc:2726] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:13:30.795878 28432 raft_consensus.cc:2454] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:30.796272 31008 leader_election.cc:302] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:13:30.796557 31346 raft_consensus.cc:2781] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.796751 31346 raft_consensus.cc:686] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:30.796993 31346 consensus_queue.cc:237] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:30.800488 27896 catalog_manager.cc:5484] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:30.823073 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:30.829080 31202 raft_consensus.cc:1261] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:30.829107 30809 raft_consensus.cc:1261] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:30.829777 31532 consensus_queue.cc:1035] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.830127 31346 consensus_queue.cc:1035] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.835189 27896 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 0f591486bd3d4b60a7d2f97bac619845 (table rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b]). Expected version 1 got 0
I20240503 14:13:30.836102 27901 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 0f591486bd3d4b60a7d2f97bac619845 (table rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b]). Expected version 1 got 0
I20240503 14:13:30.836555 31601 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:30.837333 31601 tablet.cc:1620] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:30.838732 27901 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 0f591486bd3d4b60a7d2f97bac619845 (table rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b]). Expected version 1 got 0
I20240503 14:13:30.840624 31600 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:30.841409 31600 tablet.cc:1620] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:30.848997 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b] alter complete (version 1)
I20240503 14:13:30.850093 31602 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:30.851136 31602 tablet.cc:1620] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:30.851912 27896 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 0f591486bd3d4b60a7d2f97bac619845 (table rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b]). Expected version 1 got 0
I20240503 14:13:30.952381 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:30.959504 31600 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:30.959964 31600 tablet.cc:1620] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:30.960346 31602 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:30.960952 31602 tablet.cc:1620] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:30.960991 31601 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:30.961418 31601 tablet.cc:1620] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:30.969025 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=0ddc4fcc31d2487bae6f0246af013b6b] alter complete (version 2)
I20240503 14:13:31.004418 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 14:13:31.004647 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 14:13:31.008152 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 05bdc8cd19594b4bab63c8d5ee9131c2
I20240503 14:13:31.008690 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 3ee7becef75a4016868696af456e2a3b
I20240503 14:13:31.008932 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bdc8cd19594b4bab63c8d5ee9131c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:31.009173 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bdc8cd19594b4bab63c8d5ee9131c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:31.009240 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet ea6b739f48b841e3be03047e84eae57d
I20240503 14:13:31.009474 31310 tablet_replica.cc:330] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:31.009485 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 05bdc8cd19594b4bab63c8d5ee9131c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:31.009647 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee7becef75a4016868696af456e2a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:31.009711 31310 raft_consensus.cc:2229] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.009768 31311 tablet_replica.cc:330] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:31.009855 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee7becef75a4016868696af456e2a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:31.009966 31311 raft_consensus.cc:2229] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.009989 31310 raft_consensus.cc:2258] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.010120 31543 tablet_replica.cc:330] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:31.010336 31311 raft_consensus.cc:2258] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.010356 31543 raft_consensus.cc:2229] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.010612 31306 tablet_replica.cc:330] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:31.010655 31543 raft_consensus.cc:2258] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.010777 31310 ts_tablet_manager.cc:1875] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.010841 31306 raft_consensus.cc:2229] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.011006 31311 ts_tablet_manager.cc:1875] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.011075 31306 raft_consensus.cc:2258] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.011345 31543 ts_tablet_manager.cc:1875] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.011765 31306 ts_tablet_manager.cc:1875] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.014394 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6b739f48b841e3be03047e84eae57d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:31.015228 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ee7becef75a4016868696af456e2a3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:31.015535 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6b739f48b841e3be03047e84eae57d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:31.015748 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet ea6b739f48b841e3be03047e84eae57d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:31.017920 31310 ts_tablet_manager.cc:1888] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.018460 31310 log.cc:1196] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/05bdc8cd19594b4bab63c8d5ee9131c2
I20240503 14:13:31.018694 31543 ts_tablet_manager.cc:1888] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.018930 31543 log.cc:1196] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/3ee7becef75a4016868696af456e2a3b
I20240503 14:13:31.019011 31306 ts_tablet_manager.cc:1888] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.019023 31310 ts_tablet_manager.cc:1909] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:31.019182 31306 log.cc:1196] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/05bdc8cd19594b4bab63c8d5ee9131c2
I20240503 14:13:31.019439 31543 ts_tablet_manager.cc:1909] T 3ee7becef75a4016868696af456e2a3b P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:31.019697 31306 ts_tablet_manager.cc:1909] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:31.020051 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 05bdc8cd19594b4bab63c8d5ee9131c2 (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.020138 31310 tablet_replica.cc:330] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:31.020331 31310 raft_consensus.cc:2229] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.020447 31311 ts_tablet_manager.cc:1888] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.020511 31543 tablet_replica.cc:330] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:31.020596 31311 log.cc:1196] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/05bdc8cd19594b4bab63c8d5ee9131c2
I20240503 14:13:31.020596 31310 raft_consensus.cc:2258] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.020646 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 3ee7becef75a4016868696af456e2a3b (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.020736 31543 raft_consensus.cc:2229] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.020953 31543 raft_consensus.cc:2258] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.021066 31311 ts_tablet_manager.cc:1909] T 05bdc8cd19594b4bab63c8d5ee9131c2 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:31.021286 31310 ts_tablet_manager.cc:1875] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.021565 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 05bdc8cd19594b4bab63c8d5ee9131c2 (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.021610 31543 ts_tablet_manager.cc:1875] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.021934 31306 tablet_replica.cc:330] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:31.022150 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 05bdc8cd19594b4bab63c8d5ee9131c2 (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.022195 31306 raft_consensus.cc:2229] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.022446 31306 raft_consensus.cc:2258] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.022560 31311 tablet_replica.cc:330] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:31.022840 31311 raft_consensus.cc:2229] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.023113 31311 raft_consensus.cc:2258] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.023795 31311 ts_tablet_manager.cc:1875] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.024571 31306 ts_tablet_manager.cc:1875] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.027254 31310 ts_tablet_manager.cc:1888] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.027456 31310 log.cc:1196] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/3ee7becef75a4016868696af456e2a3b
I20240503 14:13:31.027482 31543 ts_tablet_manager.cc:1888] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.027652 31543 log.cc:1196] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/ea6b739f48b841e3be03047e84eae57d
I20240503 14:13:31.027891 31310 ts_tablet_manager.cc:1909] T 3ee7becef75a4016868696af456e2a3b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:31.028120 31543 ts_tablet_manager.cc:1909] T ea6b739f48b841e3be03047e84eae57d P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:31.028861 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 3ee7becef75a4016868696af456e2a3b (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.028972 31310 tablet_replica.cc:330] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:31.029125 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet ea6b739f48b841e3be03047e84eae57d (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.029198 31310 raft_consensus.cc:2229] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.029541 31310 raft_consensus.cc:2258] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.030217 31310 ts_tablet_manager.cc:1875] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.030694 31306 ts_tablet_manager.cc:1888] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.030803 31306 log.cc:1196] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/ea6b739f48b841e3be03047e84eae57d
I20240503 14:13:31.031149 31306 ts_tablet_manager.cc:1909] T ea6b739f48b841e3be03047e84eae57d P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:31.031695 31311 ts_tablet_manager.cc:1888] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.031863 31311 log.cc:1196] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/3ee7becef75a4016868696af456e2a3b
I20240503 14:13:31.032078 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet ea6b739f48b841e3be03047e84eae57d (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.032441 31311 ts_tablet_manager.cc:1909] T 3ee7becef75a4016868696af456e2a3b P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:31.033401 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 3ee7becef75a4016868696af456e2a3b (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.035367 31310 ts_tablet_manager.cc:1888] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.035459 31310 log.cc:1196] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/ea6b739f48b841e3be03047e84eae57d
I20240503 14:13:31.035871 31310 ts_tablet_manager.cc:1909] T ea6b739f48b841e3be03047e84eae57d P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:31.036638 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet ea6b739f48b841e3be03047e84eae57d (table rebalance_test_table_16 [id=8a487903c2584d27bbd46e1279d6eb0d]) successfully deleted
I20240503 14:13:31.041268 31505 consensus_queue.cc:1035] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.047535 31505 consensus_queue.cc:1035] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.072697 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:31.080174 31600 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.081198 31601 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.081326 31602 tablet.cc:1722] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.088155 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=0ddc4fcc31d2487bae6f0246af013b6b] alter complete (version 3)
I20240503 14:13:31.182508 31392 consensus_queue.cc:1035] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.187825 31590 consensus_queue.cc:1035] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.190611 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:31.190873 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:31.193885 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 0f591486bd3d4b60a7d2f97bac619845
I20240503 14:13:31.194753 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f591486bd3d4b60a7d2f97bac619845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:31.194777 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f591486bd3d4b60a7d2f97bac619845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:31.195333 31611 tablet_replica.cc:330] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:31.195611 31611 raft_consensus.cc:2229] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.195600 31182 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f591486bd3d4b60a7d2f97bac619845 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:31.195866 31611 raft_consensus.cc:2258] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.196218 31613 tablet_replica.cc:330] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:31.196491 31613 raft_consensus.cc:2229] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.196767 31613 raft_consensus.cc:2258] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.196861 31611 ts_tablet_manager.cc:1875] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.197727 31613 ts_tablet_manager.cc:1875] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.203217 31611 ts_tablet_manager.cc:1888] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:31.203406 31611 log.cc:1196] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/0f591486bd3d4b60a7d2f97bac619845
I20240503 14:13:31.203863 31613 ts_tablet_manager.cc:1888] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:31.203938 31611 ts_tablet_manager.cc:1909] T 0f591486bd3d4b60a7d2f97bac619845 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:31.204048 31613 log.cc:1196] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/0f591486bd3d4b60a7d2f97bac619845
I20240503 14:13:31.204528 31613 ts_tablet_manager.cc:1909] T 0f591486bd3d4b60a7d2f97bac619845 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:31.205078 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 0f591486bd3d4b60a7d2f97bac619845 (table rebalancer_dynamic_table_new_name [id=0ddc4fcc31d2487bae6f0246af013b6b]) successfully deleted
I20240503 14:13:31.205214 31612 tablet_replica.cc:330] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:31.205648 31612 raft_consensus.cc:2229] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.205821 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 0f591486bd3d4b60a7d2f97bac619845 (table rebalancer_dynamic_table_new_name [id=0ddc4fcc31d2487bae6f0246af013b6b]) successfully deleted
I20240503 14:13:31.206378 31612 raft_consensus.cc:2258] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.207345 31612 ts_tablet_manager.cc:1875] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.212744 31612 ts_tablet_manager.cc:1888] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:31.212850 31612 log.cc:1196] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/0f591486bd3d4b60a7d2f97bac619845
I20240503 14:13:31.213148 31612 ts_tablet_manager.cc:1909] T 0f591486bd3d4b60a7d2f97bac619845 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:31.213971 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 0f591486bd3d4b60a7d2f97bac619845 (table rebalancer_dynamic_table_new_name [id=0ddc4fcc31d2487bae6f0246af013b6b]) successfully deleted
I20240503 14:13:31.220541 31532 consensus_queue.cc:1035] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.224562 31532 consensus_queue.cc:1035] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.296747 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:31.303583 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed7db884c11449bab8a5d2e443e1623 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1]), partition=
I20240503 14:13:31.303575 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed7db884c11449bab8a5d2e443e1623 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1]), partition=
I20240503 14:13:31.303887 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 7ed7db884c11449bab8a5d2e443e1623 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1]), partition=
I20240503 14:13:31.303990 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed7db884c11449bab8a5d2e443e1623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.303980 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed7db884c11449bab8a5d2e443e1623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.304251 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ed7db884c11449bab8a5d2e443e1623. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.312461 31619 tablet_bootstrap.cc:492] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:31.313330 31618 tablet_bootstrap.cc:492] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:31.314607 31618 tablet_bootstrap.cc:654] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.315233 31617 tablet_bootstrap.cc:492] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:31.315830 31619 tablet_bootstrap.cc:654] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.316690 31618 tablet_bootstrap.cc:492] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:31.316772 31617 tablet_bootstrap.cc:654] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.316879 31618 ts_tablet_manager.cc:1367] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:31.317446 31619 tablet_bootstrap.cc:492] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:31.317591 31619 ts_tablet_manager.cc:1367] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:31.317951 31618 raft_consensus.cc:348] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.318176 31618 raft_consensus.cc:374] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.318292 31618 raft_consensus.cc:729] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.318413 31619 raft_consensus.cc:348] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.318642 31619 raft_consensus.cc:374] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.318560 31618 consensus_queue.cc:260] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.318748 31619 raft_consensus.cc:729] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.319082 31617 tablet_bootstrap.cc:492] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:31.319244 31617 ts_tablet_manager.cc:1367] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:31.319366 31618 ts_tablet_manager.cc:1398] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:31.320084 31619 consensus_queue.cc:260] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.320145 31617 raft_consensus.cc:348] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.320340 31617 raft_consensus.cc:374] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.320456 31617 raft_consensus.cc:729] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.320709 31617 consensus_queue.cc:260] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.321313 31617 ts_tablet_manager.cc:1398] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:31.322656 31619 ts_tablet_manager.cc:1398] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:31.349982 31544 raft_consensus.cc:482] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.350185 31544 raft_consensus.cc:504] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.351092 31544 leader_election.cc:288] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:31.356155 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:31.368945 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.369431 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.370606 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 8baf804d83954d288576c313efe3d1cb (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.371130 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf804d83954d288576c313efe3d1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.372468 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.372916 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.372838 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 53432d0862e3446580b61d6eb5816a0f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.373137 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 8baf804d83954d288576c313efe3d1cb (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.373253 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53432d0862e3446580b61d6eb5816a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.373574 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf804d83954d288576c313efe3d1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.377676 31618 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:31.378641 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.379057 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.379244 31618 tablet_bootstrap.cc:654] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.381726 31617 tablet_bootstrap.cc:492] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:31.382697 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 53432d0862e3446580b61d6eb5816a0f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.383150 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53432d0862e3446580b61d6eb5816a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.383926 31617 tablet_bootstrap.cc:654] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.386803 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 8baf804d83954d288576c313efe3d1cb (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.387182 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf804d83954d288576c313efe3d1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.387449 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 53432d0862e3446580b61d6eb5816a0f (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.387868 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53432d0862e3446580b61d6eb5816a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.392614 31618 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:31.392940 31618 ts_tablet_manager.cc:1367] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240503 14:13:31.394055 31618 raft_consensus.cc:348] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.394384 31618 raft_consensus.cc:374] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.394517 31618 raft_consensus.cc:729] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.394847 31618 consensus_queue.cc:260] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.397004 31618 ts_tablet_manager.cc:1398] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:31.399228 31617 tablet_bootstrap.cc:492] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:31.399498 31617 ts_tablet_manager.cc:1367] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.009s
I20240503 14:13:31.401542 31617 raft_consensus.cc:348] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.401870 31617 raft_consensus.cc:374] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.402000 31617 raft_consensus.cc:729] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.402386 31617 consensus_queue.cc:260] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.403451 31617 ts_tablet_manager.cc:1398] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:31.403827 31617 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:31.403846 31631 tablet_bootstrap.cc:492] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:31.405637 31617 tablet_bootstrap.cc:654] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.408424 31631 tablet_bootstrap.cc:654] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.410081 31633 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:31.413204 31633 tablet_bootstrap.cc:654] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.418081 31568 raft_consensus.cc:482] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.418311 31568 raft_consensus.cc:504] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.419420 31630 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:31.419404 31568 leader_election.cc:288] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:31.420755 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf804d83954d288576c313efe3d1cb" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
W20240503 14:13:31.421556 28240 leader_election.cc:341] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.422173 31630 tablet_bootstrap.cc:654] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.424206 31631 tablet_bootstrap.cc:492] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:31.425155 31631 ts_tablet_manager.cc:1367] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240503 14:13:31.427249 31631 raft_consensus.cc:348] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.428824 31631 raft_consensus.cc:374] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.429065 31631 raft_consensus.cc:729] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.429385 31617 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:31.429589 31617 ts_tablet_manager.cc:1367] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.014s
I20240503 14:13:31.429572 31631 consensus_queue.cc:260] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.430538 31617 raft_consensus.cc:348] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.430804 31617 raft_consensus.cc:374] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.430935 31617 raft_consensus.cc:729] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.431180 31617 consensus_queue.cc:260] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.432530 31617 ts_tablet_manager.cc:1398] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:31.434291 31631 ts_tablet_manager.cc:1398] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:13:31.435443 31631 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:31.436220 31627 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:31.437896 31630 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:31.438114 31630 ts_tablet_manager.cc:1367] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 14:13:31.439034 31630 raft_consensus.cc:348] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.439308 31630 raft_consensus.cc:374] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.439467 31630 raft_consensus.cc:729] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.439818 31630 consensus_queue.cc:260] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.440135 31631 tablet_bootstrap.cc:654] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.441164 31630 ts_tablet_manager.cc:1398] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:31.441475 31630 tablet_bootstrap.cc:492] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:31.442715 31627 tablet_bootstrap.cc:654] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.443534 31633 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:31.443686 31633 ts_tablet_manager.cc:1367] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.034s user 0.015s sys 0.000s
I20240503 14:13:31.444131 31630 tablet_bootstrap.cc:654] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.444550 31633 raft_consensus.cc:348] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.446000 31633 raft_consensus.cc:374] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.446319 31630 tablet_bootstrap.cc:492] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:31.446230 31633 raft_consensus.cc:729] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.446512 31630 ts_tablet_manager.cc:1367] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 14:13:31.446565 31633 consensus_queue.cc:260] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.447196 31633 ts_tablet_manager.cc:1398] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:31.447455 31630 raft_consensus.cc:348] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.447715 31630 raft_consensus.cc:374] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.447829 31630 raft_consensus.cc:729] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.448150 31630 consensus_queue.cc:260] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.448766 31630 ts_tablet_manager.cc:1398] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:31.450611 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed7db884c11449bab8a5d2e443e1623" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:31.450940 30941 raft_consensus.cc:2454] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:31.451763 31138 leader_election.cc:302] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:31.452139 31544 raft_consensus.cc:2781] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.452239 31544 raft_consensus.cc:482] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.452338 31544 raft_consensus.cc:3037] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.455332 31544 raft_consensus.cc:504] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.457417 31473 raft_consensus.cc:482] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.457552 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed7db884c11449bab8a5d2e443e1623" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:31.457578 31473 raft_consensus.cc:504] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.457753 30941 raft_consensus.cc:3037] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.458653 31473 leader_election.cc:288] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:31.459412 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53432d0862e3446580b61d6eb5816a0f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:31.459709 31072 raft_consensus.cc:2454] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:31.460297 30745 leader_election.cc:302] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:31.460467 30941 raft_consensus.cc:2454] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:31.460858 31138 leader_election.cc:302] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:31.461202 31473 raft_consensus.cc:2781] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.461331 31473 raft_consensus.cc:482] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.461423 31473 raft_consensus.cc:3037] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.462855 31544 leader_election.cc:288] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:31.463176 31544 raft_consensus.cc:2781] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.463420 31544 raft_consensus.cc:686] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:31.464004 31544 consensus_queue.cc:237] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:31.467669 31473 raft_consensus.cc:504] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.469422 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53432d0862e3446580b61d6eb5816a0f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:31.469692 31072 raft_consensus.cc:3037] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.473075 31072 raft_consensus.cc:2454] T 53432d0862e3446580b61d6eb5816a0f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:31.474483 30745 leader_election.cc:302] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:31.475091 31645 raft_consensus.cc:2781] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.477447 27899 catalog_manager.cc:5484] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:31.478158 31473 leader_election.cc:288] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:31.484346 31631 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:31.485241 31631 ts_tablet_manager.cc:1367] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20240503 14:13:31.486663 31631 raft_consensus.cc:348] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.487888 31631 raft_consensus.cc:374] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.488168 31631 raft_consensus.cc:729] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.488791 31631 consensus_queue.cc:260] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.491211 31631 ts_tablet_manager.cc:1398] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20240503 14:13:31.492501 28350 tablet.cc:2369] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:31.495222 31627 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:31.495398 31627 ts_tablet_manager.cc:1367] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.017s
I20240503 14:13:31.496330 31627 raft_consensus.cc:348] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.496577 31627 raft_consensus.cc:374] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.496670 31627 raft_consensus.cc:729] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.496928 31627 consensus_queue.cc:260] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.498481 31645 raft_consensus.cc:686] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:31.498801 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed7db884c11449bab8a5d2e443e1623" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:31.498867 31645 consensus_queue.cc:237] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:31.499078 28303 raft_consensus.cc:2454] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:31.499464 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ed7db884c11449bab8a5d2e443e1623" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:31.499650 28304 raft_consensus.cc:3037] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.503000 31627 ts_tablet_manager.cc:1398] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 14:13:31.503027 28304 raft_consensus.cc:2454] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:31.503778 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf804d83954d288576c313efe3d1cb" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:31.503994 31202 raft_consensus.cc:2454] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:31.504825 28238 leader_election.cc:302] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, a4cdf05256a0407c8929ac43eb55d44f; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:13:31.505254 31568 raft_consensus.cc:2781] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.505378 31568 raft_consensus.cc:482] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.505486 31568 raft_consensus.cc:3037] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.507884 31568 raft_consensus.cc:504] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.509547 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf804d83954d288576c313efe3d1cb" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:31.509743 28040 raft_consensus.cc:3037] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.511951 28040 raft_consensus.cc:2454] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:31.512341 28240 leader_election.cc:302] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:31.512646 31616 raft_consensus.cc:2781] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.513082 31568 leader_election.cc:288] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:31.513021 27899 catalog_manager.cc:5484] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:31.513932 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf804d83954d288576c313efe3d1cb" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:31.514192 31202 raft_consensus.cc:3037] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.516921 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 14:13:31.517165 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 14:13:31.520536 31202 raft_consensus.cc:2454] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:31.521090 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet fa1b4e565b1244a39a0428646dcb0013
I20240503 14:13:31.521628 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet d909ff34d9624201a1085eeb8169200f
I20240503 14:13:31.522053 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet eb0209b00d0e4ee1ac975a60ab500f71
I20240503 14:13:31.523674 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet d909ff34d9624201a1085eeb8169200f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:31.523912 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1b4e565b1244a39a0428646dcb0013 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:31.524026 28284 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0209b00d0e4ee1ac975a60ab500f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:31.524106 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet d909ff34d9624201a1085eeb8169200f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:31.524201 31310 tablet_replica.cc:330] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:31.524293 31543 tablet_replica.cc:330] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:31.524420 31310 raft_consensus.cc:2229] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.524539 31543 raft_consensus.cc:2229] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.524650 31310 raft_consensus.cc:2258] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.524770 31543 raft_consensus.cc:2258] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.525017 28153 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0209b00d0e4ee1ac975a60ab500f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:31.525272 31311 tablet_replica.cc:330] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:31.525418 31310 ts_tablet_manager.cc:1875] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.525483 31543 ts_tablet_manager.cc:1875] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.525532 31311 raft_consensus.cc:2229] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.525733 31311 raft_consensus.cc:2258] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.524164 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet d909ff34d9624201a1085eeb8169200f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:31.526430 31311 ts_tablet_manager.cc:1875] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.526549 28022 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0209b00d0e4ee1ac975a60ab500f71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:31.529978 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1b4e565b1244a39a0428646dcb0013 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:31.530393 31306 tablet_replica.cc:330] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:31.530633 31306 raft_consensus.cc:2229] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.530850 31306 raft_consensus.cc:2258] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.531533 31306 ts_tablet_manager.cc:1875] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.533124 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53432d0862e3446580b61d6eb5816a0f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:31.533147 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53432d0862e3446580b61d6eb5816a0f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:31.533442 31202 raft_consensus.cc:2454] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:31.533748 31311 ts_tablet_manager.cc:1888] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.533903 31311 log.cc:1196] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/eb0209b00d0e4ee1ac975a60ab500f71
I20240503 14:13:31.534060 31310 ts_tablet_manager.cc:1888] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.534193 31310 log.cc:1196] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/fa1b4e565b1244a39a0428646dcb0013
I20240503 14:13:31.534358 31311 ts_tablet_manager.cc:1909] T eb0209b00d0e4ee1ac975a60ab500f71 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:31.524020 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1b4e565b1244a39a0428646dcb0013 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:31.534606 31310 ts_tablet_manager.cc:1909] T fa1b4e565b1244a39a0428646dcb0013 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:31.535256 31311 tablet_replica.cc:330] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:31.535337 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet eb0209b00d0e4ee1ac975a60ab500f71 (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.535460 31311 raft_consensus.cc:2229] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.535656 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet fa1b4e565b1244a39a0428646dcb0013 (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.535735 31310 tablet_replica.cc:330] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:31.535776 31311 raft_consensus.cc:2258] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.535928 31310 raft_consensus.cc:2229] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.536223 31310 raft_consensus.cc:2258] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.536394 31311 ts_tablet_manager.cc:1875] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.536866 31310 ts_tablet_manager.cc:1875] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.539249 31543 ts_tablet_manager.cc:1888] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.539381 31543 log.cc:1196] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb0209b00d0e4ee1ac975a60ab500f71
I20240503 14:13:31.539729 31543 ts_tablet_manager.cc:1909] T eb0209b00d0e4ee1ac975a60ab500f71 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:31.539826 31532 raft_consensus.cc:482] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.540004 31532 raft_consensus.cc:504] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.540743 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet eb0209b00d0e4ee1ac975a60ab500f71 (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.540856 31532 leader_election.cc:288] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:31.541401 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1fbc1d1bb6439c9f8b64364ad9ab4b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:31.541635 30809 raft_consensus.cc:2454] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:31.542047 31005 leader_election.cc:302] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:31.542727 31310 ts_tablet_manager.cc:1888] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.542907 31310 log.cc:1196] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/eb0209b00d0e4ee1ac975a60ab500f71
I20240503 14:13:31.543061 31532 raft_consensus.cc:2781] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.543219 31532 raft_consensus.cc:482] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.543357 31532 raft_consensus.cc:3037] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.543527 31310 ts_tablet_manager.cc:1909] T eb0209b00d0e4ee1ac975a60ab500f71 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:31.544598 31310 tablet_replica.cc:330] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:31.544900 31310 raft_consensus.cc:2229] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.545199 31310 raft_consensus.cc:2258] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.545804 31310 ts_tablet_manager.cc:1875] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.546422 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet eb0209b00d0e4ee1ac975a60ab500f71 (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.547545 31311 ts_tablet_manager.cc:1888] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.547714 31311 log.cc:1196] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/fa1b4e565b1244a39a0428646dcb0013
I20240503 14:13:31.548235 31311 ts_tablet_manager.cc:1909] T fa1b4e565b1244a39a0428646dcb0013 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:31.549515 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet fa1b4e565b1244a39a0428646dcb0013 (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.550465 31532 raft_consensus.cc:504] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.552011 31310 ts_tablet_manager.cc:1888] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.552114 31306 ts_tablet_manager.cc:1888] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.552268 31310 log.cc:1196] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/d909ff34d9624201a1085eeb8169200f
I20240503 14:13:31.552277 31306 log.cc:1196] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/d909ff34d9624201a1085eeb8169200f
I20240503 14:13:31.552706 31306 ts_tablet_manager.cc:1909] T d909ff34d9624201a1085eeb8169200f P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:31.552723 31310 ts_tablet_manager.cc:1909] T d909ff34d9624201a1085eeb8169200f P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:31.552583 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1fbc1d1bb6439c9f8b64364ad9ab4b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:31.552968 30809 raft_consensus.cc:3037] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.553364 31543 tablet_replica.cc:330] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:31.553576 31543 raft_consensus.cc:2229] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.553848 31543 raft_consensus.cc:2258] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.554484 31543 ts_tablet_manager.cc:1875] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.556636 30809 raft_consensus.cc:2454] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:31.557034 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet d909ff34d9624201a1085eeb8169200f (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.557482 31005 leader_election.cc:302] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:31.557773 31346 raft_consensus.cc:2781] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.557951 31346 raft_consensus.cc:686] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:31.558303 31616 raft_consensus.cc:686] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:31.558209 31346 consensus_queue.cc:237] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:31.558604 31616 consensus_queue.cc:237] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:31.561688 31543 ts_tablet_manager.cc:1888] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.561806 31543 log.cc:1196] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/d909ff34d9624201a1085eeb8169200f
I20240503 14:13:31.562312 31532 leader_election.cc:288] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:31.563576 31543 ts_tablet_manager.cc:1909] T d909ff34d9624201a1085eeb8169200f P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:31.564893 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet d909ff34d9624201a1085eeb8169200f (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.565083 31543 tablet_replica.cc:330] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:31.565358 31543 raft_consensus.cc:2229] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.565634 31543 raft_consensus.cc:2258] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.566440 31543 ts_tablet_manager.cc:1875] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.566491 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet d909ff34d9624201a1085eeb8169200f (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.567869 27896 catalog_manager.cc:5484] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:31.568552 27899 catalog_manager.cc:5484] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:31.572800 31543 ts_tablet_manager.cc:1888] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:31.572996 31543 log.cc:1196] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/fa1b4e565b1244a39a0428646dcb0013
I20240503 14:13:31.573438 31543 ts_tablet_manager.cc:1909] T fa1b4e565b1244a39a0428646dcb0013 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:31.573963 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1fbc1d1bb6439c9f8b64364ad9ab4b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:31.573908 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e1fbc1d1bb6439c9f8b64364ad9ab4b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:31.574144 28172 raft_consensus.cc:3037] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.575018 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet fa1b4e565b1244a39a0428646dcb0013 (table rebalance_test_table_18 [id=66565a6bba6741c98b5a90323d8041b9]) successfully deleted
I20240503 14:13:31.576319 28172 raft_consensus.cc:2454] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:31.613226 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:31.636966 30941 raft_consensus.cc:1261] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.636930 28304 raft_consensus.cc:1261] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.637622 31643 consensus_queue.cc:1035] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.638124 31544 consensus_queue.cc:1035] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.653985 27896 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 7ed7db884c11449bab8a5d2e443e1623 (table rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1]). Expected version 1 got 0
I20240503 14:13:31.654800 31654 mvcc.cc:205] Tried to move back new op lower bound from 7023598025258614784 to 7023598024562356224. Current Snapshot: MvccSnapshot[applied={T|T < 7023598025258614784}]
I20240503 14:13:31.655602 27899 catalog_manager.cc:5576] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889) does not have the latest schema for tablet 7ed7db884c11449bab8a5d2e443e1623 (table rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1]). Expected version 1 got 0
I20240503 14:13:31.658262 31659 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:31.658932 31659 tablet.cc:1620] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:31.663600 31660 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:31.664377 31660 tablet.cc:1620] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:31.667411 27896 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 7ed7db884c11449bab8a5d2e443e1623 (table rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1]). Expected version 1 got 0
I20240503 14:13:31.667833 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1] alter complete (version 1)
I20240503 14:13:31.667374 31661 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:31.668838 27901 catalog_manager.cc:5576] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889) does not have the latest schema for tablet 7ed7db884c11449bab8a5d2e443e1623 (table rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1]). Expected version 1 got 0
I20240503 14:13:31.669360 31661 tablet.cc:1620] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:31.778898 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:31.785722 31659 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:31.786303 31661 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:31.786437 31659 tablet.cc:1620] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:31.787026 31661 tablet.cc:1620] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:31.788127 31660 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:31.788668 31660 tablet.cc:1620] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:31.798259 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=c062b2c5a69a40c989661a811acaaca1] alter complete (version 2)
I20240503 14:13:31.902093 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:31.904413 31202 raft_consensus.cc:3037] T 53432d0862e3446580b61d6eb5816a0f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.906921 31645 consensus_queue.cc:1035] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.911623 31660 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.915683 31659 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.916419 31646 consensus_queue.cc:1035] T 53432d0862e3446580b61d6eb5816a0f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.917892 31661 tablet.cc:1722] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.925096 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=c062b2c5a69a40c989661a811acaaca1] alter complete (version 3)
I20240503 14:13:31.967432 31616 consensus_queue.cc:1035] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.971758 31616 consensus_queue.cc:1035] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.029697 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:32.029960 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:32.032858 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 7ed7db884c11449bab8a5d2e443e1623
I20240503 14:13:32.033599 31181 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ed7db884c11449bab8a5d2e443e1623 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:32.033890 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ed7db884c11449bab8a5d2e443e1623 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:32.034030 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ed7db884c11449bab8a5d2e443e1623 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:32.034195 31543 tablet_replica.cc:330] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:32.034444 31543 raft_consensus.cc:2229] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.034572 31669 tablet_replica.cc:330] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:32.034695 31543 raft_consensus.cc:2258] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.034814 31669 raft_consensus.cc:2229] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:32.035343 31669 raft_consensus.cc:2258] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.035856 31543 ts_tablet_manager.cc:1875] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:32.036237 31669 ts_tablet_manager.cc:1875] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:32.042203 31669 ts_tablet_manager.cc:1888] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:32.042387 31669 log.cc:1196] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/7ed7db884c11449bab8a5d2e443e1623
I20240503 14:13:32.042579 31543 ts_tablet_manager.cc:1888] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:32.042750 31543 log.cc:1196] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/7ed7db884c11449bab8a5d2e443e1623
I20240503 14:13:32.042904 31669 ts_tablet_manager.cc:1909] T 7ed7db884c11449bab8a5d2e443e1623 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:32.043218 31543 ts_tablet_manager.cc:1909] T 7ed7db884c11449bab8a5d2e443e1623 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:32.043991 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 7ed7db884c11449bab8a5d2e443e1623 (table rebalancer_dynamic_table_new_name [id=c062b2c5a69a40c989661a811acaaca1]) successfully deleted
I20240503 14:13:32.044093 31670 tablet_replica.cc:330] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:32.044349 31670 raft_consensus.cc:2229] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.044361 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 7ed7db884c11449bab8a5d2e443e1623 (table rebalancer_dynamic_table_new_name [id=c062b2c5a69a40c989661a811acaaca1]) successfully deleted
I20240503 14:13:32.044656 31670 raft_consensus.cc:2258] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.045336 31670 ts_tablet_manager.cc:1875] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:32.048691 31346 consensus_queue.cc:1035] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.051313 31346 consensus_queue.cc:1035] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.052264 31670 ts_tablet_manager.cc:1888] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:32.052454 31670 log.cc:1196] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/7ed7db884c11449bab8a5d2e443e1623
I20240503 14:13:32.053120 31670 ts_tablet_manager.cc:1909] T 7ed7db884c11449bab8a5d2e443e1623 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:32.054611 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 7ed7db884c11449bab8a5d2e443e1623 (table rebalancer_dynamic_table_new_name [id=c062b2c5a69a40c989661a811acaaca1]) successfully deleted
I20240503 14:13:32.105115 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:32.113646 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 02559d1ab4824298ba76a44a7f05303b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.113853 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 02559d1ab4824298ba76a44a7f05303b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.113929 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 02559d1ab4824298ba76a44a7f05303b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.114037 31182 tablet_service.cc:1467] Processing CreateTablet for tablet f7e304f721e447bc8c4cc6eec3db872b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.114159 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02559d1ab4824298ba76a44a7f05303b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.114352 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02559d1ab4824298ba76a44a7f05303b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.114429 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02559d1ab4824298ba76a44a7f05303b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.116147 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e304f721e447bc8c4cc6eec3db872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.119644 28283 tablet_service.cc:1467] Processing CreateTablet for tablet f7e304f721e447bc8c4cc6eec3db872b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.120105 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e304f721e447bc8c4cc6eec3db872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.122165 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 0abfcd38ecc64acfac35c676b941581a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.122699 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abfcd38ecc64acfac35c676b941581a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.127072 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 0abfcd38ecc64acfac35c676b941581a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.127195 31052 tablet_service.cc:1467] Processing CreateTablet for tablet f7e304f721e447bc8c4cc6eec3db872b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.127502 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abfcd38ecc64acfac35c676b941581a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.127622 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7e304f721e447bc8c4cc6eec3db872b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.127208 31674 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:32.129451 31674 tablet_bootstrap.cc:654] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.137756 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 0abfcd38ecc64acfac35c676b941581a (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.138370 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abfcd38ecc64acfac35c676b941581a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.139338 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:32.142915 31677 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:32.150003 31677 tablet_bootstrap.cc:654] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.150523 31676 tablet_bootstrap.cc:492] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:32.157518 31681 tablet_bootstrap.cc:492] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:32.159260 31677 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:32.159477 31677 ts_tablet_manager.cc:1367] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240503 14:13:32.159749 31681 tablet_bootstrap.cc:654] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.160318 31677 raft_consensus.cc:348] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.160640 31677 raft_consensus.cc:374] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.160780 31677 raft_consensus.cc:729] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.161279 31677 consensus_queue.cc:260] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.161636 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 7995a9fba792406abe1d2e9536e0d42f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6]), partition=
I20240503 14:13:32.161994 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7995a9fba792406abe1d2e9536e0d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.162088 31677 ts_tablet_manager.cc:1398] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:32.162369 31677 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:32.164949 31677 tablet_bootstrap.cc:654] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.166731 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 7995a9fba792406abe1d2e9536e0d42f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6]), partition=
I20240503 14:13:32.167117 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7995a9fba792406abe1d2e9536e0d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.168818 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 7995a9fba792406abe1d2e9536e0d42f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6]), partition=
I20240503 14:13:32.169188 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7995a9fba792406abe1d2e9536e0d42f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.173346 31677 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:32.173449 31674 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:32.173607 31677 ts_tablet_manager.cc:1367] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20240503 14:13:32.173677 31674 ts_tablet_manager.cc:1367] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.016s
I20240503 14:13:32.174597 31677 raft_consensus.cc:348] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.174968 31679 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:32.175397 31677 raft_consensus.cc:374] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.175540 31677 raft_consensus.cc:729] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.175853 31677 consensus_queue.cc:260] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.176525 31677 ts_tablet_manager.cc:1398] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:32.177491 31676 tablet_bootstrap.cc:654] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.180891 31674 raft_consensus.cc:348] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.181234 31674 raft_consensus.cc:374] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.181416 31674 raft_consensus.cc:729] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.181771 31674 consensus_queue.cc:260] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.184582 31679 tablet_bootstrap.cc:654] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.191217 31676 tablet_bootstrap.cc:492] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:32.192026 31676 ts_tablet_manager.cc:1367] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20240503 14:13:32.193202 31676 raft_consensus.cc:348] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.194911 31676 raft_consensus.cc:374] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.196074 31676 raft_consensus.cc:729] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.196352 31674 ts_tablet_manager.cc:1398] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.023s user 0.004s sys 0.003s
I20240503 14:13:32.196501 31676 consensus_queue.cc:260] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.197470 31674 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:32.198642 31676 ts_tablet_manager.cc:1398] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:13:32.199311 31674 tablet_bootstrap.cc:654] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.203274 31679 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:32.203565 31679 ts_tablet_manager.cc:1367] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240503 14:13:32.204912 31679 raft_consensus.cc:348] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.205302 31679 raft_consensus.cc:374] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.205458 31679 raft_consensus.cc:729] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.205835 31679 consensus_queue.cc:260] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.207783 31679 ts_tablet_manager.cc:1398] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:32.207993 31392 raft_consensus.cc:482] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.208174 31392 raft_consensus.cc:504] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.208828 31674 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:32.208986 31674 ts_tablet_manager.cc:1367] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240503 14:13:32.209627 31674 raft_consensus.cc:348] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.209833 31674 raft_consensus.cc:374] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.209903 31674 raft_consensus.cc:729] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.210458 31685 tablet_bootstrap.cc:492] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:32.211967 31685 tablet_bootstrap.cc:654] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.212375 31674 consensus_queue.cc:260] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.213805 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02559d1ab4824298ba76a44a7f05303b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:32.214057 30941 raft_consensus.cc:2454] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:32.214221 31674 ts_tablet_manager.cc:1398] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:32.214527 31674 tablet_bootstrap.cc:492] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:32.215131 31392 leader_election.cc:288] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:32.215525 31679 tablet_bootstrap.cc:492] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:32.216010 31674 tablet_bootstrap.cc:654] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.216409 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02559d1ab4824298ba76a44a7f05303b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:32.216657 31202 raft_consensus.cc:2454] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:32.217212 28108 leader_election.cc:302] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:32.217674 31679 tablet_bootstrap.cc:654] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.224102 31392 raft_consensus.cc:2781] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:32.224251 31392 raft_consensus.cc:482] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:32.224329 31681 tablet_bootstrap.cc:492] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:32.224360 31392 raft_consensus.cc:3037] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.224603 31681 ts_tablet_manager.cc:1367] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.067s user 0.018s sys 0.026s
I20240503 14:13:32.225584 31681 raft_consensus.cc:348] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.225872 31681 raft_consensus.cc:374] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.226015 31681 raft_consensus.cc:729] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.226413 31681 consensus_queue.cc:260] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.226683 31685 tablet_bootstrap.cc:492] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:32.226882 31685 ts_tablet_manager.cc:1367] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240503 14:13:32.227703 31685 raft_consensus.cc:348] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.227952 31685 raft_consensus.cc:374] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.228117 31685 raft_consensus.cc:729] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.228343 31681 ts_tablet_manager.cc:1398] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:32.228402 31685 consensus_queue.cc:260] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.228695 31681 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:32.230698 31685 ts_tablet_manager.cc:1398] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:32.230885 31681 tablet_bootstrap.cc:654] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.232558 31674 tablet_bootstrap.cc:492] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:32.232744 31674 ts_tablet_manager.cc:1367] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20240503 14:13:32.233436 31679 tablet_bootstrap.cc:492] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:32.233655 31679 ts_tablet_manager.cc:1367] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240503 14:13:32.233652 31674 raft_consensus.cc:348] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.233942 31674 raft_consensus.cc:374] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.234067 31674 raft_consensus.cc:729] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.234438 31674 consensus_queue.cc:260] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.234517 31679 raft_consensus.cc:348] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.234818 31679 raft_consensus.cc:374] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.234964 31679 raft_consensus.cc:729] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.235152 31674 ts_tablet_manager.cc:1398] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:32.235303 31679 consensus_queue.cc:260] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.236007 31679 ts_tablet_manager.cc:1398] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:32.236377 31679 tablet_bootstrap.cc:492] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:32.238230 31679 tablet_bootstrap.cc:654] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.238404 31392 raft_consensus.cc:504] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.241086 31392 leader_election.cc:288] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:32.242266 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02559d1ab4824298ba76a44a7f05303b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:32.242499 31202 raft_consensus.cc:3037] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.244412 31681 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:32.244551 31681 ts_tablet_manager.cc:1367] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.003s
I20240503 14:13:32.245484 31681 raft_consensus.cc:348] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.245720 31681 raft_consensus.cc:374] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.245792 31681 raft_consensus.cc:729] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.246060 31681 consensus_queue.cc:260] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.247458 31681 ts_tablet_manager.cc:1398] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:32.248394 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02559d1ab4824298ba76a44a7f05303b" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:32.248617 30941 raft_consensus.cc:3037] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.249049 31202 raft_consensus.cc:2454] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:32.251873 31679 tablet_bootstrap.cc:492] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:32.252015 31679 ts_tablet_manager.cc:1367] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240503 14:13:32.252430 28108 leader_election.cc:302] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:32.252759 31679 raft_consensus.cc:348] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.252938 31679 raft_consensus.cc:374] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.253016 31679 raft_consensus.cc:729] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.253212 31689 raft_consensus.cc:2781] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:32.253211 31679 consensus_queue.cc:260] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.253353 31689 raft_consensus.cc:686] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:32.253782 31679 ts_tablet_manager.cc:1398] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:32.253651 31689 consensus_queue.cc:237] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:32.254608 30941 raft_consensus.cc:2454] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:32.258378 27894 catalog_manager.cc:5484] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:32.456681 31658 raft_consensus.cc:482] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.456873 31658 raft_consensus.cc:504] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.457731 31658 leader_election.cc:288] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:32.459089 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995a9fba792406abe1d2e9536e0d42f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:32.459338 30941 raft_consensus.cc:2454] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:32.459710 31138 leader_election.cc:302] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:32.460008 31658 raft_consensus.cc:2781] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:32.460145 31658 raft_consensus.cc:482] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:32.460233 31658 raft_consensus.cc:3037] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.462529 31658 raft_consensus.cc:504] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.463593 31658 leader_election.cc:288] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:32.463625 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995a9fba792406abe1d2e9536e0d42f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:32.463815 30941 raft_consensus.cc:3037] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.464890 31690 raft_consensus.cc:482] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.465039 31690 raft_consensus.cc:504] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.465850 31690 leader_election.cc:288] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:32.466192 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:32.466459 30940 raft_consensus.cc:2454] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:32.466523 30941 raft_consensus.cc:2454] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:32.466487 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995a9fba792406abe1d2e9536e0d42f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:32.466488 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7995a9fba792406abe1d2e9536e0d42f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:32.466733 28040 raft_consensus.cc:2454] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:32.467231 31138 leader_election.cc:302] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:32.467841 28109 leader_election.cc:302] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:32.468005 31658 raft_consensus.cc:2781] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:32.468101 31690 raft_consensus.cc:2781] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:32.468220 31690 raft_consensus.cc:482] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:32.468309 31690 raft_consensus.cc:3037] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.468562 31658 raft_consensus.cc:686] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:32.468997 31658 consensus_queue.cc:237] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:32.470575 31690 raft_consensus.cc:504] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.471500 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:32.471654 31690 leader_election.cc:288] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:32.471874 30941 raft_consensus.cc:3037] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.473019 27896 catalog_manager.cc:5484] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:32.474660 30941 raft_consensus.cc:2454] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:32.475157 28109 leader_election.cc:302] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:32.475509 31689 raft_consensus.cc:2781] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:32.475659 31532 raft_consensus.cc:482] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.475716 31689 raft_consensus.cc:686] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:32.475843 31532 raft_consensus.cc:504] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.476001 31689 consensus_queue.cc:237] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:32.476588 31532 leader_election.cc:288] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:32.477627 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:32.477852 30940 raft_consensus.cc:2379] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fde134d0fa2b4900b9badd2e08fea09d in current term 1: Already voted for candidate 6409961ee8c0445687c11724213c44a9 in this term.
I20240503 14:13:32.478297 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:32.479096 31006 leader_election.cc:302] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:13:32.479703 31532 raft_consensus.cc:3037] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.483765 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:32.483740 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:32.484159 31532 raft_consensus.cc:2726] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:32.486796 31071 raft_consensus.cc:2454] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:32.487346 27896 catalog_manager.cc:5484] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:32.511889 31532 raft_consensus.cc:482] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.512096 31532 raft_consensus.cc:504] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.512686 31532 leader_election.cc:288] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:32.513155 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e304f721e447bc8c4cc6eec3db872b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:32.513160 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e304f721e447bc8c4cc6eec3db872b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:32.513381 31203 raft_consensus.cc:2454] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:32.513393 28304 raft_consensus.cc:2454] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:32.513926 31008 leader_election.cc:302] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:32.514314 31532 raft_consensus.cc:2781] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:32.514436 31532 raft_consensus.cc:482] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:32.514533 31532 raft_consensus.cc:3037] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.516722 31532 raft_consensus.cc:504] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.517246 31532 leader_election.cc:288] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:32.517611 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e304f721e447bc8c4cc6eec3db872b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:32.517691 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7e304f721e447bc8c4cc6eec3db872b" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:32.517768 31203 raft_consensus.cc:3037] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.517871 28304 raft_consensus.cc:3037] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.519716 31203 raft_consensus.cc:2454] T f7e304f721e447bc8c4cc6eec3db872b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:32.519809 28304 raft_consensus.cc:2454] T f7e304f721e447bc8c4cc6eec3db872b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:32.520109 31006 leader_election.cc:302] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:32.520355 31532 raft_consensus.cc:2781] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:32.520495 31532 raft_consensus.cc:686] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:32.520798 31532 consensus_queue.cc:237] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:32.523422 27896 catalog_manager.cc:5484] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:32.603884 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:32.610673 28040 raft_consensus.cc:3037] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.611292 30941 raft_consensus.cc:1261] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:32.612076 31701 consensus_queue.cc:1035] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.613772 28040 raft_consensus.cc:1261] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:32.614434 31701 consensus_queue.cc:1035] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.617050 27896 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 7995a9fba792406abe1d2e9536e0d42f (table rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6]). Expected version 1 got 0
I20240503 14:13:32.617151 31709 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:32.620301 31709 tablet.cc:1620] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:32.621094 27896 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 7995a9fba792406abe1d2e9536e0d42f (table rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6]). Expected version 1 got 0
I20240503 14:13:32.624722 31710 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:32.625392 31710 tablet.cc:1620] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:32.628769 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6] alter complete (version 1)
I20240503 14:13:32.631647 27901 catalog_manager.cc:5576] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293) does not have the latest schema for tablet 7995a9fba792406abe1d2e9536e0d42f (table rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6]). Expected version 1 got 0
I20240503 14:13:32.655261 31713 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:32.656051 31713 tablet.cc:1620] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:32.657330 27896 catalog_manager.cc:5576] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293) does not have the latest schema for tablet 7995a9fba792406abe1d2e9536e0d42f (table rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6]). Expected version 1 got 0
I20240503 14:13:32.659693 31690 consensus_queue.cc:1035] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.666543 31690 consensus_queue.cc:1035] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.738103 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:32.745357 31709 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:32.745941 31709 tablet.cc:1620] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:32.745980 31713 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:32.746441 31713 tablet.cc:1620] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:32.746790 31710 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:32.747264 31710 tablet.cc:1620] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:32.755746 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=41d5fe0dada847f797442c1c8da92fe6] alter complete (version 2)
I20240503 14:13:32.859019 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:32.866971 31709 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:32.868397 31710 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:32.877039 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=41d5fe0dada847f797442c1c8da92fe6] alter complete (version 3)
I20240503 14:13:32.882105 31713 tablet.cc:1722] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:32.968149 31689 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.972692 31690 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.982793 27901 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:32.982981 27901 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:32.985739 27901 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 7995a9fba792406abe1d2e9536e0d42f
I20240503 14:13:32.986505 31182 tablet_service.cc:1514] Processing DeleteTablet for tablet 7995a9fba792406abe1d2e9536e0d42f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:32.986706 31532 consensus_queue.cc:1035] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.986773 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 7995a9fba792406abe1d2e9536e0d42f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:32.987113 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 7995a9fba792406abe1d2e9536e0d42f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:32.997205 31723 tablet_replica.cc:330] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:32.997350 31722 tablet_replica.cc:330] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:32.997603 31723 raft_consensus.cc:2229] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.997622 31722 raft_consensus.cc:2229] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:32.997655 31724 tablet_replica.cc:330] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:32.997865 31724 raft_consensus.cc:2229] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.997874 31723 raft_consensus.cc:2258] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.998054 31722 raft_consensus.cc:2258] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.998173 31724 raft_consensus.cc:2258] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.998836 31723 ts_tablet_manager.cc:1875] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:32.998962 31722 ts_tablet_manager.cc:1875] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:32.999191 31724 ts_tablet_manager.cc:1875] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.001978 31346 consensus_queue.cc:1035] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.005183 31722 ts_tablet_manager.cc:1888] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.005306 31722 log.cc:1196] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/7995a9fba792406abe1d2e9536e0d42f
I20240503 14:13:33.005712 31722 ts_tablet_manager.cc:1909] T 7995a9fba792406abe1d2e9536e0d42f P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:33.006906 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 7995a9fba792406abe1d2e9536e0d42f (table rebalancer_dynamic_table_new_name [id=41d5fe0dada847f797442c1c8da92fe6]) successfully deleted
I20240503 14:13:33.008752 31723 ts_tablet_manager.cc:1888] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.008881 31723 log.cc:1196] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/7995a9fba792406abe1d2e9536e0d42f
I20240503 14:13:33.009371 31723 ts_tablet_manager.cc:1909] T 7995a9fba792406abe1d2e9536e0d42f P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:33.010336 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 7995a9fba792406abe1d2e9536e0d42f (table rebalancer_dynamic_table_new_name [id=41d5fe0dada847f797442c1c8da92fe6]) successfully deleted
I20240503 14:13:33.011833 31724 ts_tablet_manager.cc:1888] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.011979 31724 log.cc:1196] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/7995a9fba792406abe1d2e9536e0d42f
I20240503 14:13:33.012404 31724 ts_tablet_manager.cc:1909] T 7995a9fba792406abe1d2e9536e0d42f P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:33.013341 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 7995a9fba792406abe1d2e9536e0d42f (table rebalancer_dynamic_table_new_name [id=41d5fe0dada847f797442c1c8da92fe6]) successfully deleted
I20240503 14:13:33.036334 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:33.044934 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1c93fac26f4e0092c5806fe1aef59b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.045436 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1c93fac26f4e0092c5806fe1aef59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.046766 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 5815f4c96eef4130aa0582d33a24b391 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.046926 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 5815f4c96eef4130aa0582d33a24b391 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.047180 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5815f4c96eef4130aa0582d33a24b391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.047358 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5815f4c96eef4130aa0582d33a24b391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.050628 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1c93fac26f4e0092c5806fe1aef59b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.050706 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 5815f4c96eef4130aa0582d33a24b391 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.051064 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1c93fac26f4e0092c5806fe1aef59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.051138 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5815f4c96eef4130aa0582d33a24b391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.054591 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 361b2d07d8b44d128acfb3cc941ac01a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.054710 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 361b2d07d8b44d128acfb3cc941ac01a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.055071 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361b2d07d8b44d128acfb3cc941ac01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.055195 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361b2d07d8b44d128acfb3cc941ac01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.059473 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 361b2d07d8b44d128acfb3cc941ac01a (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.060002 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361b2d07d8b44d128acfb3cc941ac01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.061398 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 5e1c93fac26f4e0092c5806fe1aef59b (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.061820 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e1c93fac26f4e0092c5806fe1aef59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.066030 31728 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:33.067051 31730 tablet_bootstrap.cc:492] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:33.067878 31728 tablet_bootstrap.cc:654] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.069000 31730 tablet_bootstrap.cc:654] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.073292 31729 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:33.077453 31729 tablet_bootstrap.cc:654] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.081511 31736 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:33.083559 31730 tablet_bootstrap.cc:492] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:33.083977 31730 ts_tablet_manager.cc:1367] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20240503 14:13:33.084538 31736 tablet_bootstrap.cc:654] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.085036 31730 raft_consensus.cc:348] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.085348 31730 raft_consensus.cc:374] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.085502 31730 raft_consensus.cc:729] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.085847 31730 consensus_queue.cc:260] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.086428 31729 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:33.086644 31732 tablet_bootstrap.cc:492] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:33.086700 31730 ts_tablet_manager.cc:1398] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.088308 31729 ts_tablet_manager.cc:1367] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20240503 14:13:33.089107 31729 raft_consensus.cc:348] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.089749 31734 tablet_bootstrap.cc:492] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:33.090000 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:33.091527 31734 tablet_bootstrap.cc:654] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.093930 31732 tablet_bootstrap.cc:654] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.094296 31729 raft_consensus.cc:374] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.094461 31729 raft_consensus.cc:729] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.094769 31729 consensus_queue.cc:260] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.099009 31729 ts_tablet_manager.cc:1398] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 14:13:33.100488 31729 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:33.101171 30920 tablet_service.cc:1467] Processing CreateTablet for tablet c786fd8d199d457b9820fba791c3daf4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c]), partition=
I20240503 14:13:33.101454 31051 tablet_service.cc:1467] Processing CreateTablet for tablet c786fd8d199d457b9820fba791c3daf4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c]), partition=
I20240503 14:13:33.101533 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c786fd8d199d457b9820fba791c3daf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.101778 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c786fd8d199d457b9820fba791c3daf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.102686 30790 tablet_service.cc:1467] Processing CreateTablet for tablet c786fd8d199d457b9820fba791c3daf4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c]), partition=
I20240503 14:13:33.102990 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c786fd8d199d457b9820fba791c3daf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.107440 31729 tablet_bootstrap.cc:654] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.110977 31736 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:33.111578 31736 ts_tablet_manager.cc:1367] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.009s
I20240503 14:13:33.112515 31736 raft_consensus.cc:348] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.113346 31729 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:33.113516 31729 ts_tablet_manager.cc:1367] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:13:33.113943 31736 raft_consensus.cc:374] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.114073 31734 tablet_bootstrap.cc:492] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:33.114063 31736 raft_consensus.cc:729] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.114223 31734 ts_tablet_manager.cc:1367] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240503 14:13:33.114171 31729 raft_consensus.cc:348] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.114418 31729 raft_consensus.cc:374] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.114548 31729 raft_consensus.cc:729] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.114588 31736 consensus_queue.cc:260] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.114975 31729 consensus_queue.cc:260] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.115181 31734 raft_consensus.cc:348] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.115437 31734 raft_consensus.cc:374] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.115566 31734 raft_consensus.cc:729] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.115847 31734 consensus_queue.cc:260] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.116340 31729 ts_tablet_manager.cc:1398] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.117262 31732 tablet_bootstrap.cc:492] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:33.117431 31732 ts_tablet_manager.cc:1367] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20240503 14:13:33.118083 31728 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:33.118230 31728 ts_tablet_manager.cc:1367] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.012s
I20240503 14:13:33.119092 31732 raft_consensus.cc:348] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.119174 31728 raft_consensus.cc:348] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.119426 31728 raft_consensus.cc:374] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.119458 31732 raft_consensus.cc:374] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.119530 31728 raft_consensus.cc:729] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.119649 31732 raft_consensus.cc:729] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.119755 31728 consensus_queue.cc:260] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.120002 31732 consensus_queue.cc:260] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.120865 31646 raft_consensus.cc:482] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.121126 31728 ts_tablet_manager.cc:1398] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.121024 31646 raft_consensus.cc:504] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.122082 31646 leader_election.cc:288] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:33.122764 31728 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:33.122713 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:33.122928 31071 raft_consensus.cc:2454] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:33.123332 31732 ts_tablet_manager.cc:1398] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:13:33.124217 31732 tablet_bootstrap.cc:492] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:33.124317 30745 leader_election.cc:302] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:33.125663 31732 tablet_bootstrap.cc:654] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.126470 31736 ts_tablet_manager.cc:1398] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 14:13:33.126566 31646 raft_consensus.cc:2781] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.126657 31646 raft_consensus.cc:482] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.126765 31646 raft_consensus.cc:3037] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.127996 31736 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:33.129936 31736 tablet_bootstrap.cc:654] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.131428 31728 tablet_bootstrap.cc:654] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.133702 31732 tablet_bootstrap.cc:492] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:33.133957 31732 ts_tablet_manager.cc:1367] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 14:13:33.134721 31732 raft_consensus.cc:348] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.135385 31732 raft_consensus.cc:374] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.135517 31732 raft_consensus.cc:729] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.135804 31732 consensus_queue.cc:260] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.137391 31732 ts_tablet_manager.cc:1398] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:33.142515 31717 raft_consensus.cc:482] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.142699 31717 raft_consensus.cc:504] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.143332 31717 leader_election.cc:288] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:33.143478 31736 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:33.143659 31736 ts_tablet_manager.cc:1367] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20240503 14:13:33.143886 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:33.145133 31736 raft_consensus.cc:348] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.145409 31736 raft_consensus.cc:374] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.145488 31736 raft_consensus.cc:729] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.145754 31736 consensus_queue.cc:260] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.146462 31736 ts_tablet_manager.cc:1398] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.147697 31728 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:33.147912 31728 ts_tablet_manager.cc:1367] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.002s
I20240503 14:13:33.148770 31728 raft_consensus.cc:348] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.149047 31728 raft_consensus.cc:374] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.149168 31728 raft_consensus.cc:729] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.149433 31728 consensus_queue.cc:260] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.150367 31728 ts_tablet_manager.cc:1398] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:33.150727 31728 tablet_bootstrap.cc:492] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:33.151248 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:33.151492 31071 raft_consensus.cc:2454] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:33.152377 31728 tablet_bootstrap.cc:654] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.152902 28239 leader_election.cc:302] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:33.157105 31646 raft_consensus.cc:504] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.157809 31717 raft_consensus.cc:2781] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.157951 31717 raft_consensus.cc:482] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.158048 31717 raft_consensus.cc:3037] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.158354 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:33.158483 30809 raft_consensus.cc:2379] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4cdf05256a0407c8929ac43eb55d44f in current term 1: Already voted for candidate d0dbfb0e43f948c3922926aacc365fc1 in this term.
I20240503 14:13:33.160161 31734 ts_tablet_manager.cc:1398] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.046s user 0.000s sys 0.002s
I20240503 14:13:33.160455 31734 tablet_bootstrap.cc:492] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:33.160861 31717 raft_consensus.cc:504] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.161332 28304 raft_consensus.cc:2379] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0dbfb0e43f948c3922926aacc365fc1 in current term 1: Already voted for candidate a4cdf05256a0407c8929ac43eb55d44f in this term.
I20240503 14:13:33.161644 31717 leader_election.cc:288] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:33.162163 31734 tablet_bootstrap.cc:654] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.164085 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:33.164455 30809 raft_consensus.cc:2379] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a4cdf05256a0407c8929ac43eb55d44f in current term 1: Already voted for candidate d0dbfb0e43f948c3922926aacc365fc1 in this term.
I20240503 14:13:33.165729 31646 leader_election.cc:288] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:33.166299 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:33.166601 31071 raft_consensus.cc:3037] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.168150 31728 tablet_bootstrap.cc:492] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:33.168309 31728 ts_tablet_manager.cc:1367] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20240503 14:13:33.169107 31728 raft_consensus.cc:348] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.169335 31728 raft_consensus.cc:374] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.169432 31728 raft_consensus.cc:729] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.169551 31071 raft_consensus.cc:2454] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:33.169685 31728 consensus_queue.cc:260] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.171094 31728 ts_tablet_manager.cc:1398] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:33.171782 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:33.172052 31072 raft_consensus.cc:2379] T 5e1c93fac26f4e0092c5806fe1aef59b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a4cdf05256a0407c8929ac43eb55d44f in current term 1: Already voted for candidate d0dbfb0e43f948c3922926aacc365fc1 in this term.
I20240503 14:13:33.169965 30745 leader_election.cc:302] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:33.172477 28239 leader_election.cc:302] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f; no voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:33.172616 31646 raft_consensus.cc:2781] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.173090 31717 raft_consensus.cc:2726] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:13:33.173206 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e1c93fac26f4e0092c5806fe1aef59b" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:33.173432 28304 raft_consensus.cc:2379] T 5e1c93fac26f4e0092c5806fe1aef59b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d0dbfb0e43f948c3922926aacc365fc1 in current term 1: Already voted for candidate a4cdf05256a0407c8929ac43eb55d44f in this term.
I20240503 14:13:33.174683 31734 tablet_bootstrap.cc:492] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:33.175176 31734 ts_tablet_manager.cc:1367] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240503 14:13:33.175496 31686 raft_consensus.cc:482] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.175629 31686 raft_consensus.cc:504] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.176294 31686 leader_election.cc:288] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:33.176697 31734 raft_consensus.cc:348] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.176908 31646 raft_consensus.cc:686] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:33.176970 31734 raft_consensus.cc:374] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.177037 31734 raft_consensus.cc:729] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.177175 31646 consensus_queue.cc:237] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:33.177278 31734 consensus_queue.cc:260] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.177942 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c786fd8d199d457b9820fba791c3daf4" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:33.178169 31072 raft_consensus.cc:2454] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.178525 31734 ts_tablet_manager.cc:1398] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.178866 30876 leader_election.cc:302] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:33.179649 31686 raft_consensus.cc:2781] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.179751 31686 raft_consensus.cc:482] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.179819 31686 raft_consensus.cc:3037] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.179832 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c786fd8d199d457b9820fba791c3daf4" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:33.180006 30809 raft_consensus.cc:2454] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.182911 31686 raft_consensus.cc:504] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.182834 27894 catalog_manager.cc:5484] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:33.183707 31686 leader_election.cc:288] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:33.183841 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c786fd8d199d457b9820fba791c3daf4" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:33.183948 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c786fd8d199d457b9820fba791c3daf4" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:33.184015 30809 raft_consensus.cc:3037] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.184132 31072 raft_consensus.cc:3037] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.185950 30809 raft_consensus.cc:2454] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.186445 30874 leader_election.cc:302] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:33.186698 31686 raft_consensus.cc:2781] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.187131 31686 raft_consensus.cc:686] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:33.188205 31686 consensus_queue.cc:237] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:33.188563 31072 raft_consensus.cc:2454] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.192739 27901 catalog_manager.cc:5484] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:33.214586 31686 raft_consensus.cc:482] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.214776 31686 raft_consensus.cc:504] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.215363 31686 leader_election.cc:288] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:33.215741 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5815f4c96eef4130aa0582d33a24b391" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:33.215786 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5815f4c96eef4130aa0582d33a24b391" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:33.215952 28040 raft_consensus.cc:2454] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.216004 31203 raft_consensus.cc:2454] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.216342 30877 leader_election.cc:302] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:33.216627 31686 raft_consensus.cc:2781] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.216734 31686 raft_consensus.cc:482] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.216861 31686 raft_consensus.cc:3037] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.218967 31686 raft_consensus.cc:504] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.219520 31686 leader_election.cc:288] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:33.219892 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5815f4c96eef4130aa0582d33a24b391" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:33.219892 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5815f4c96eef4130aa0582d33a24b391" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:33.220043 28040 raft_consensus.cc:3037] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.220068 31203 raft_consensus.cc:3037] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.221954 31203 raft_consensus.cc:2454] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.221971 28040 raft_consensus.cc:2454] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.222335 30877 leader_election.cc:302] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:33.222613 31686 raft_consensus.cc:2781] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.222808 31686 raft_consensus.cc:686] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:33.223059 31686 consensus_queue.cc:237] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.225786 27901 catalog_manager.cc:5484] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:33.311980 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:33.318084 31072 raft_consensus.cc:1261] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.318493 30809 raft_consensus.cc:1261] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.318703 31686 consensus_queue.cc:1035] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.319092 31751 consensus_queue.cc:1035] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.322718 31710 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:33.322962 27901 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet c786fd8d199d457b9820fba791c3daf4 (table rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c]). Expected version 1 got 0
I20240503 14:13:33.323185 31710 tablet.cc:1620] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:33.324121 31757 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:33.324640 31757 tablet.cc:1620] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:33.326323 31758 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:33.327045 31758 tablet.cc:1620] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:33.328045 27894 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet c786fd8d199d457b9820fba791c3daf4 (table rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c]). Expected version 1 got 0
I20240503 14:13:33.332571 27894 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet c786fd8d199d457b9820fba791c3daf4 (table rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c]). Expected version 1 got 0
I20240503 14:13:33.334363 31686 raft_consensus.cc:482] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.334525 31686 raft_consensus.cc:504] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.335099 31686 leader_election.cc:288] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:33.335392 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c] alter complete (version 1)
I20240503 14:13:33.335676 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:33.335945 31203 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.336395 30875 leader_election.cc:302] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:33.336376 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:33.336604 28040 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.336669 31686 raft_consensus.cc:2781] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.336746 31686 raft_consensus.cc:482] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.336831 31686 raft_consensus.cc:3037] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.339748 31686 raft_consensus.cc:504] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.340513 31686 leader_election.cc:288] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:33.340709 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:33.340893 31203 raft_consensus.cc:3037] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.342443 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:33.342669 28040 raft_consensus.cc:3037] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.343021 31203 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.343429 30875 leader_election.cc:302] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:33.343680 31686 raft_consensus.cc:2781] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.343839 31686 raft_consensus.cc:686] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:33.344158 31686 consensus_queue.cc:237] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.345332 28040 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.361425 27894 catalog_manager.cc:5484] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:33.440104 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:33.447026 31710 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:33.447600 31710 tablet.cc:1620] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:33.447677 31757 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:33.448158 31757 tablet.cc:1620] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:33.453441 31758 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:33.453938 31758 tablet.cc:1620] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:33.456305 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=06aca6c0bc64419f81ce239cb52fa81c] alter complete (version 2)
I20240503 14:13:33.558924 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:33.567230 31710 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:33.568543 31757 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:33.568675 31758 tablet.cc:1722] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:33.576668 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=06aca6c0bc64419f81ce239cb52fa81c] alter complete (version 3)
I20240503 14:13:33.670851 31646 consensus_queue.cc:1035] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.676728 31747 consensus_queue.cc:1035] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.680084 27896 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:33.680303 27896 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:33.682948 31686 consensus_queue.cc:1035] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.683221 27896 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet c786fd8d199d457b9820fba791c3daf4
I20240503 14:13:33.683943 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet c786fd8d199d457b9820fba791c3daf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:33 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:33.684700 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet c786fd8d199d457b9820fba791c3daf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:33 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:33.685282 31769 tablet_replica.cc:330] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:33.685568 31769 raft_consensus.cc:2229] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:33.685595 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet c786fd8d199d457b9820fba791c3daf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:33 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:33.685906 31769 raft_consensus.cc:2258] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.686076 31768 tablet_replica.cc:330] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:33.686326 31768 raft_consensus.cc:2229] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:33.686722 31768 raft_consensus.cc:2258] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.687099 31769 ts_tablet_manager.cc:1875] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.687649 31768 ts_tablet_manager.cc:1875] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.691768 31770 tablet_replica.cc:330] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:33.694532 31769 ts_tablet_manager.cc:1888] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.694685 31769 log.cc:1196] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/c786fd8d199d457b9820fba791c3daf4
I20240503 14:13:33.694837 31768 ts_tablet_manager.cc:1888] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.694967 31768 log.cc:1196] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/c786fd8d199d457b9820fba791c3daf4
I20240503 14:13:33.695150 31769 ts_tablet_manager.cc:1909] T c786fd8d199d457b9820fba791c3daf4 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:33.695389 31768 ts_tablet_manager.cc:1909] T c786fd8d199d457b9820fba791c3daf4 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:33.695426 31751 consensus_queue.cc:1035] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.696247 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet c786fd8d199d457b9820fba791c3daf4 (table rebalancer_dynamic_table_new_name [id=06aca6c0bc64419f81ce239cb52fa81c]) successfully deleted
I20240503 14:13:33.696496 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet c786fd8d199d457b9820fba791c3daf4 (table rebalancer_dynamic_table_new_name [id=06aca6c0bc64419f81ce239cb52fa81c]) successfully deleted
I20240503 14:13:33.697248 31770 raft_consensus.cc:2229] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:33.698534 31770 raft_consensus.cc:2258] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.699481 31770 ts_tablet_manager.cc:1875] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.704761 31770 ts_tablet_manager.cc:1888] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.704885 31770 log.cc:1196] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/c786fd8d199d457b9820fba791c3daf4
I20240503 14:13:33.705333 31770 ts_tablet_manager.cc:1909] T c786fd8d199d457b9820fba791c3daf4 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:33.706225 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet c786fd8d199d457b9820fba791c3daf4 (table rebalancer_dynamic_table_new_name [id=06aca6c0bc64419f81ce239cb52fa81c]) successfully deleted
I20240503 14:13:33.779194 31686 consensus_queue.cc:1035] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.783775 31762 consensus_queue.cc:1035] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.785476 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:33.793792 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 0b039476f702459ea20ad63466b8eb58 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7]), partition=
I20240503 14:13:33.794010 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 0b039476f702459ea20ad63466b8eb58 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7]), partition=
I20240503 14:13:33.794420 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b039476f702459ea20ad63466b8eb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.794673 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 0b039476f702459ea20ad63466b8eb58 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7]), partition=
I20240503 14:13:33.794996 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b039476f702459ea20ad63466b8eb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.800688 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b039476f702459ea20ad63466b8eb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.800947 31777 tablet_bootstrap.cc:492] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:33.803165 31777 tablet_bootstrap.cc:654] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.805917 31780 tablet_bootstrap.cc:492] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:33.806844 31778 tablet_bootstrap.cc:492] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:33.807224 31777 tablet_bootstrap.cc:492] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:33.807379 31777 ts_tablet_manager.cc:1367] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:13:33.807574 31780 tablet_bootstrap.cc:654] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.808238 31777 raft_consensus.cc:348] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.808498 31777 raft_consensus.cc:374] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.808549 31778 tablet_bootstrap.cc:654] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.808666 31777 raft_consensus.cc:729] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.809011 31777 consensus_queue.cc:260] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.809635 31780 tablet_bootstrap.cc:492] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:33.809803 31777 ts_tablet_manager.cc:1398] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:33.809824 31780 ts_tablet_manager.cc:1367] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:33.810531 31780 raft_consensus.cc:348] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.811206 31780 raft_consensus.cc:374] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.811359 31780 raft_consensus.cc:729] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.811972 31778 tablet_bootstrap.cc:492] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:33.812178 31778 ts_tablet_manager.cc:1367] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:33.812173 31780 consensus_queue.cc:260] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.812736 31780 ts_tablet_manager.cc:1398] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:33.813170 31778 raft_consensus.cc:348] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.813460 31778 raft_consensus.cc:374] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.813582 31778 raft_consensus.cc:729] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.813823 31778 consensus_queue.cc:260] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.814565 31778 ts_tablet_manager.cc:1398] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:33.878769 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:33.888691 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 710e09d5ab4e457fbbf54b3acaf5ff15 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.888978 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 710e09d5ab4e457fbbf54b3acaf5ff15 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.889189 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710e09d5ab4e457fbbf54b3acaf5ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.889328 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 710e09d5ab4e457fbbf54b3acaf5ff15 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.889457 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710e09d5ab4e457fbbf54b3acaf5ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.889740 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710e09d5ab4e457fbbf54b3acaf5ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.893952 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 98a0a9b88c6a4d8ba55e2aa4662dcdbe (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.894397 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a0a9b88c6a4d8ba55e2aa4662dcdbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.894619 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 98a0a9b88c6a4d8ba55e2aa4662dcdbe (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.895056 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a0a9b88c6a4d8ba55e2aa4662dcdbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.895124 31778 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:33.895282 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 98a0a9b88c6a4d8ba55e2aa4662dcdbe (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.895615 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98a0a9b88c6a4d8ba55e2aa4662dcdbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.896725 31778 tablet_bootstrap.cc:654] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.898658 31051 tablet_service.cc:1467] Processing CreateTablet for tablet d58ff895fc8e4ddb84e2c1db165799d1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.899070 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58ff895fc8e4ddb84e2c1db165799d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.902052 30789 tablet_service.cc:1467] Processing CreateTablet for tablet d58ff895fc8e4ddb84e2c1db165799d1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.902441 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58ff895fc8e4ddb84e2c1db165799d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.893952 30921 tablet_service.cc:1467] Processing CreateTablet for tablet d58ff895fc8e4ddb84e2c1db165799d1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.902768 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58ff895fc8e4ddb84e2c1db165799d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.912297 31784 tablet_bootstrap.cc:492] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:33.914129 31784 tablet_bootstrap.cc:654] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.914172 31777 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:33.917371 31777 tablet_bootstrap.cc:654] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.918720 31778 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:33.919009 31778 ts_tablet_manager.cc:1367] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20240503 14:13:33.922482 31778 raft_consensus.cc:348] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.922780 31778 raft_consensus.cc:374] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.922865 31778 raft_consensus.cc:729] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.923197 31778 consensus_queue.cc:260] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.923652 31780 tablet_bootstrap.cc:492] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:33.926757 31780 tablet_bootstrap.cc:654] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.928627 31788 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:33.930949 31788 tablet_bootstrap.cc:654] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.933595 31780 tablet_bootstrap.cc:492] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:33.934396 31780 ts_tablet_manager.cc:1367] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240503 14:13:33.935026 31780 raft_consensus.cc:348] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.935232 31780 raft_consensus.cc:374] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.935319 31780 raft_consensus.cc:729] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.932271 31778 ts_tablet_manager.cc:1398] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:33.935596 31785 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:33.936934 31784 tablet_bootstrap.cc:492] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:33.937088 31784 ts_tablet_manager.cc:1367] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.004s
I20240503 14:13:33.937147 31785 tablet_bootstrap.cc:654] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.937865 31784 raft_consensus.cc:348] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.938047 31784 raft_consensus.cc:374] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.938148 31784 raft_consensus.cc:729] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.938222 31780 consensus_queue.cc:260] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.939693 31780 ts_tablet_manager.cc:1398] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:33.939978 31780 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:33.940706 31784 consensus_queue.cc:260] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.942646 31784 ts_tablet_manager.cc:1398] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:33.942957 31777 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:33.943024 31784 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:33.943188 31777 ts_tablet_manager.cc:1367] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.029s user 0.019s sys 0.007s
I20240503 14:13:33.944048 31777 raft_consensus.cc:348] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.944373 31777 raft_consensus.cc:374] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.944509 31777 raft_consensus.cc:729] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.944710 31784 tablet_bootstrap.cc:654] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.944844 31777 consensus_queue.cc:260] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.945325 31780 tablet_bootstrap.cc:654] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.945712 31777 ts_tablet_manager.cc:1398] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:33.947489 31777 tablet_bootstrap.cc:492] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:33.949638 31777 tablet_bootstrap.cc:654] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.950599 31788 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:33.950825 31788 ts_tablet_manager.cc:1367] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240503 14:13:33.952029 31780 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:33.952011 31788 raft_consensus.cc:348] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.952365 31788 raft_consensus.cc:374] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.952538 31788 raft_consensus.cc:729] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.952558 31780 ts_tablet_manager.cc:1367] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20240503 14:13:33.952913 31788 consensus_queue.cc:260] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.953526 31780 raft_consensus.cc:348] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.953799 31780 raft_consensus.cc:374] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.953922 31780 raft_consensus.cc:729] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.954680 31788 ts_tablet_manager.cc:1398] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.955889 31785 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:33.956082 31785 ts_tablet_manager.cc:1367] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240503 14:13:33.956578 31780 consensus_queue.cc:260] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:33.957926 31785 raft_consensus.cc:348] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.958220 31785 raft_consensus.cc:374] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.958397 31785 raft_consensus.cc:729] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.958750 31785 consensus_queue.cc:260] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.959483 31785 ts_tablet_manager.cc:1398] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.963609 31777 tablet_bootstrap.cc:492] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:33.963719 31784 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:33.963804 31777 ts_tablet_manager.cc:1367] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.012s
I20240503 14:13:33.963917 31784 ts_tablet_manager.cc:1367] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.007s
I20240503 14:13:33.965584 31777 raft_consensus.cc:348] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.965875 31777 raft_consensus.cc:374] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.965799 31784 raft_consensus.cc:348] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.966024 31777 raft_consensus.cc:729] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.966074 31784 raft_consensus.cc:374] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.966197 31784 raft_consensus.cc:729] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.966342 31777 consensus_queue.cc:260] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:33.966547 31784 consensus_queue.cc:260] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.967195 31784 ts_tablet_manager.cc:1398] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:33.967654 31780 ts_tablet_manager.cc:1398] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 14:13:33.968405 31777 ts_tablet_manager.cc:1398] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:33.980894 31549 raft_consensus.cc:482] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.981066 31549 raft_consensus.cc:504] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.981712 31549 leader_election.cc:288] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:33.982153 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710e09d5ab4e457fbbf54b3acaf5ff15" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:33.982225 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710e09d5ab4e457fbbf54b3acaf5ff15" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:33.982406 28040 raft_consensus.cc:2454] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:33.982498 31203 raft_consensus.cc:2454] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:33.982827 28370 leader_election.cc:302] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:33.983116 31549 raft_consensus.cc:2781] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.983209 31549 raft_consensus.cc:482] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.983321 31549 raft_consensus.cc:3037] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.984490 31762 raft_consensus.cc:482] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.984655 31762 raft_consensus.cc:504] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.985304 31762 leader_election.cc:288] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:33.985304 31549 raft_consensus.cc:504] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.985785 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b039476f702459ea20ad63466b8eb58" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:33.985822 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b039476f702459ea20ad63466b8eb58" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:33.986043 30809 raft_consensus.cc:2454] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.986027 31203 raft_consensus.cc:2454] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:33.986508 30874 leader_election.cc:302] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:33.986519 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710e09d5ab4e457fbbf54b3acaf5ff15" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:33.986716 31203 raft_consensus.cc:3037] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.986657 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "710e09d5ab4e457fbbf54b3acaf5ff15" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:33.986809 31762 raft_consensus.cc:2781] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.986858 28040 raft_consensus.cc:3037] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.986910 31762 raft_consensus.cc:482] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.987018 31762 raft_consensus.cc:3037] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.989049 31762 raft_consensus.cc:504] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.989526 31549 leader_election.cc:288] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:33.989642 28040 raft_consensus.cc:2454] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:33.989822 31762 leader_election.cc:288] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:33.989869 31203 raft_consensus.cc:2454] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:33.990114 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b039476f702459ea20ad63466b8eb58" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:33.990288 28368 leader_election.cc:302] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:13:33.990372 30809 raft_consensus.cc:3037] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.990353 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b039476f702459ea20ad63466b8eb58" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:33.990561 31202 raft_consensus.cc:3037] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.990662 31549 raft_consensus.cc:2781] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.990893 31549 raft_consensus.cc:686] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:33.991160 31549 consensus_queue.cc:237] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.993142 31202 raft_consensus.cc:2454] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.993170 30809 raft_consensus.cc:2454] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:33.993496 30875 leader_election.cc:302] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:33.993750 31762 raft_consensus.cc:2781] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.993959 31762 raft_consensus.cc:686] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:33.994228 31762 consensus_queue.cc:237] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:33.995528 27901 catalog_manager.cc:5484] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:33.997804 27894 catalog_manager.cc:5484] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:34.016156 31747 raft_consensus.cc:482] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.016335 31747 raft_consensus.cc:504] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.016897 31747 leader_election.cc:288] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:34.017379 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58ff895fc8e4ddb84e2c1db165799d1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:34.017467 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58ff895fc8e4ddb84e2c1db165799d1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:34.017591 31072 raft_consensus.cc:2454] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:34.017697 30940 raft_consensus.cc:2454] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:34.018064 30745 leader_election.cc:302] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:34.018343 31747 raft_consensus.cc:2781] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.018450 31747 raft_consensus.cc:482] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.018551 31747 raft_consensus.cc:3037] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.020601 31747 raft_consensus.cc:504] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.021104 31747 leader_election.cc:288] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:34.021756 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58ff895fc8e4ddb84e2c1db165799d1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:34.021871 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58ff895fc8e4ddb84e2c1db165799d1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:34.021976 31072 raft_consensus.cc:3037] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.022056 30940 raft_consensus.cc:3037] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.023736 31072 raft_consensus.cc:2454] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:34.023851 30940 raft_consensus.cc:2454] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:34.024135 30745 leader_election.cc:302] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:34.024376 31747 raft_consensus.cc:2781] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.024540 31747 raft_consensus.cc:686] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:34.024793 31747 consensus_queue.cc:237] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.027784 27894 catalog_manager.cc:5484] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:34.054196 31762 raft_consensus.cc:482] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.054417 31762 raft_consensus.cc:504] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.054975 31762 leader_election.cc:288] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:34.055424 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a0a9b88c6a4d8ba55e2aa4662dcdbe" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:34.055495 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a0a9b88c6a4d8ba55e2aa4662dcdbe" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:34.055670 28040 raft_consensus.cc:2454] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:34.055711 30809 raft_consensus.cc:2454] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:34.056092 30877 leader_election.cc:302] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:34.056351 31762 raft_consensus.cc:2781] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.056457 31762 raft_consensus.cc:482] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.056576 31762 raft_consensus.cc:3037] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.059125 31762 raft_consensus.cc:504] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.059726 31762 leader_election.cc:288] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:34.060166 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a0a9b88c6a4d8ba55e2aa4662dcdbe" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:34.060421 28040 raft_consensus.cc:3037] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.060401 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a0a9b88c6a4d8ba55e2aa4662dcdbe" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:34.060606 30809 raft_consensus.cc:3037] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.063297 28040 raft_consensus.cc:2454] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:34.063388 30809 raft_consensus.cc:2454] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:34.063695 30877 leader_election.cc:302] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:34.064141 31762 raft_consensus.cc:2781] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.064291 31762 raft_consensus.cc:686] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:34.064595 31762 consensus_queue.cc:237] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.067734 27894 catalog_manager.cc:5484] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:34.106072 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:34.111268 31202 raft_consensus.cc:1261] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.111379 30809 raft_consensus.cc:1261] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.111980 31762 consensus_queue.cc:1035] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.112283 31686 consensus_queue.cc:1035] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.117128 27894 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 0b039476f702459ea20ad63466b8eb58 (table rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7]). Expected version 1 got 0
I20240503 14:13:34.117257 31801 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.117846 31801 tablet.cc:1620] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:34.118175 27901 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 0b039476f702459ea20ad63466b8eb58 (table rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7]). Expected version 1 got 0
I20240503 14:13:34.118374 31802 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.119032 31802 tablet.cc:1620] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:34.124928 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7] alter complete (version 1)
I20240503 14:13:34.125319 31803 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.126492 31803 tablet.cc:1620] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:34.127360 27896 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 0b039476f702459ea20ad63466b8eb58 (table rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7]). Expected version 1 got 0
I20240503 14:13:34.228175 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:34.235380 31801 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:34.235634 31802 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:34.235845 31801 tablet.cc:1620] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:34.236130 31802 tablet.cc:1620] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:34.236137 31803 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:34.236649 31803 tablet.cc:1620] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:34.244444 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=4eb1fcc6ff07490f97954c3b5ba55be7] alter complete (version 2)
I20240503 14:13:34.346169 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:34.355746 31802 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:34.357429 31803 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:34.362406 31801 tablet.cc:1722] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:34.370585 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=4eb1fcc6ff07490f97954c3b5ba55be7] alter complete (version 3)
I20240503 14:13:34.407780 31747 consensus_queue.cc:1035] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.412560 31749 consensus_queue.cc:1035] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.455456 31761 consensus_queue.cc:1035] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.464231 31812 consensus_queue.cc:1035] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.472201 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:34.472401 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:34.475291 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 0b039476f702459ea20ad63466b8eb58
I20240503 14:13:34.476181 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b039476f702459ea20ad63466b8eb58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:34 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:34.476341 31181 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b039476f702459ea20ad63466b8eb58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:34 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:34.476301 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b039476f702459ea20ad63466b8eb58 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:34 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:34.489094 31817 tablet_replica.cc:330] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:34.489277 31816 tablet_replica.cc:330] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:34.489339 31817 raft_consensus.cc:2229] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:34.489485 31816 raft_consensus.cc:2229] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:34.489698 31816 raft_consensus.cc:2258] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:34.489786 31817 raft_consensus.cc:2258] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:34.490628 31816 ts_tablet_manager.cc:1875] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:34.490651 31817 ts_tablet_manager.cc:1875] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:34.497078 31816 ts_tablet_manager.cc:1888] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:34.497284 31816 log.cc:1196] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/0b039476f702459ea20ad63466b8eb58
I20240503 14:13:34.497356 31817 ts_tablet_manager.cc:1888] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:34.497520 31817 log.cc:1196] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/0b039476f702459ea20ad63466b8eb58
I20240503 14:13:34.498070 31817 ts_tablet_manager.cc:1909] T 0b039476f702459ea20ad63466b8eb58 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:34.498070 31816 ts_tablet_manager.cc:1909] T 0b039476f702459ea20ad63466b8eb58 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:34.499115 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 0b039476f702459ea20ad63466b8eb58 (table rebalancer_dynamic_table_new_name [id=4eb1fcc6ff07490f97954c3b5ba55be7]) successfully deleted
I20240503 14:13:34.499279 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 0b039476f702459ea20ad63466b8eb58 (table rebalancer_dynamic_table_new_name [id=4eb1fcc6ff07490f97954c3b5ba55be7]) successfully deleted
I20240503 14:13:34.502712 31815 tablet_replica.cc:330] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:34.503048 31815 raft_consensus.cc:2229] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:34.503265 31815 raft_consensus.cc:2258] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:34.504091 31815 ts_tablet_manager.cc:1875] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:34.510149 31815 ts_tablet_manager.cc:1888] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:34.510288 31815 log.cc:1196] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/0b039476f702459ea20ad63466b8eb58
I20240503 14:13:34.510708 31815 ts_tablet_manager.cc:1909] T 0b039476f702459ea20ad63466b8eb58 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:34.511658 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 0b039476f702459ea20ad63466b8eb58 (table rebalancer_dynamic_table_new_name [id=4eb1fcc6ff07490f97954c3b5ba55be7]) successfully deleted
I20240503 14:13:34.523293 31762 consensus_queue.cc:1035] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.527307 31686 consensus_queue.cc:1035] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.578680 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:34.585214 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 83675aee0d0d4b1794275c4684c30ca0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670]), partition=
I20240503 14:13:34.585304 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 83675aee0d0d4b1794275c4684c30ca0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670]), partition=
I20240503 14:13:34.585604 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83675aee0d0d4b1794275c4684c30ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.585614 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83675aee0d0d4b1794275c4684c30ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.591064 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 83675aee0d0d4b1794275c4684c30ca0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670]), partition=
I20240503 14:13:34.591135 31820 tablet_bootstrap.cc:492] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:34.591490 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83675aee0d0d4b1794275c4684c30ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.592330 31821 tablet_bootstrap.cc:492] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:34.592751 31820 tablet_bootstrap.cc:654] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.593811 31821 tablet_bootstrap.cc:654] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.594761 31820 tablet_bootstrap.cc:492] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:34.594988 31820 ts_tablet_manager.cc:1367] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20240503 14:13:34.597820 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:34.595885 31820 raft_consensus.cc:348] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.600759 31820 raft_consensus.cc:374] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.600905 31820 raft_consensus.cc:729] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.603725 31824 tablet_bootstrap.cc:492] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:34.605796 31824 tablet_bootstrap.cc:654] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.609030 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 7b82b7a5fb7c4e13b126f6d47164a463 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.609441 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b82b7a5fb7c4e13b126f6d47164a463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.609593 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 618ba33946424065a465217bae7cdbb0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.610023 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ba33946424065a465217bae7cdbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.609938 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 7b82b7a5fb7c4e13b126f6d47164a463 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.610343 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b82b7a5fb7c4e13b126f6d47164a463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.601209 31820 consensus_queue.cc:260] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.614648 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 618ba33946424065a465217bae7cdbb0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.615065 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ba33946424065a465217bae7cdbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.614668 31820 ts_tablet_manager.cc:1398] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240503 14:13:34.617864 31052 tablet_service.cc:1467] Processing CreateTablet for tablet e57c283674c342169940a5903516ec93 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.618391 31821 tablet_bootstrap.cc:492] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:34.618427 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 618ba33946424065a465217bae7cdbb0 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.618547 31821 ts_tablet_manager.cc:1367] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.006s
I20240503 14:13:34.618795 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57c283674c342169940a5903516ec93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.619539 31821 raft_consensus.cc:348] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.619765 31821 raft_consensus.cc:374] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.619861 31821 raft_consensus.cc:729] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.619935 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 7b82b7a5fb7c4e13b126f6d47164a463 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.620383 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b82b7a5fb7c4e13b126f6d47164a463. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.620780 31821 consensus_queue.cc:260] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.622076 31821 ts_tablet_manager.cc:1398] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:34.622546 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ba33946424065a465217bae7cdbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.636003 30921 tablet_service.cc:1467] Processing CreateTablet for tablet e57c283674c342169940a5903516ec93 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.636490 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57c283674c342169940a5903516ec93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.637051 31821 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:34.637126 31828 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:34.638769 31821 tablet_bootstrap.cc:654] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.640817 30789 tablet_service.cc:1467] Processing CreateTablet for tablet e57c283674c342169940a5903516ec93 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:34.644094 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57c283674c342169940a5903516ec93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.644901 31828 tablet_bootstrap.cc:654] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.646592 31824 tablet_bootstrap.cc:492] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:34.646823 31824 ts_tablet_manager.cc:1367] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20240503 14:13:34.647833 31824 raft_consensus.cc:348] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.648731 31824 raft_consensus.cc:374] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.648938 31824 raft_consensus.cc:729] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.649376 31824 consensus_queue.cc:260] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.651515 31824 ts_tablet_manager.cc:1398] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:34.652395 31827 tablet_bootstrap.cc:492] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:34.652915 31824 tablet_bootstrap.cc:492] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:34.657866 31828 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:34.657982 31824 tablet_bootstrap.cc:654] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.658143 31828 ts_tablet_manager.cc:1367] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20240503 14:13:34.658990 31828 raft_consensus.cc:348] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.659288 31828 raft_consensus.cc:374] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.659466 31828 raft_consensus.cc:729] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.660019 31828 consensus_queue.cc:260] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.661088 31827 tablet_bootstrap.cc:654] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.661823 31828 ts_tablet_manager.cc:1398] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:34.662590 31826 tablet_bootstrap.cc:492] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:34.663429 31821 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:34.663638 31821 ts_tablet_manager.cc:1367] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20240503 14:13:34.664664 31821 raft_consensus.cc:348] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.665004 31821 raft_consensus.cc:374] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.665158 31821 raft_consensus.cc:729] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.665470 31821 consensus_queue.cc:260] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.666421 31821 ts_tablet_manager.cc:1398] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:34.667955 31821 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:34.667954 31826 tablet_bootstrap.cc:654] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.668675 31820 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:34.666736 31831 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:34.670266 31831 tablet_bootstrap.cc:654] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.670267 31820 tablet_bootstrap.cc:654] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.671823 31824 tablet_bootstrap.cc:492] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:34.672374 31824 ts_tablet_manager.cc:1367] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20240503 14:13:34.673542 31824 raft_consensus.cc:348] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.674667 31824 raft_consensus.cc:374] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.674199 31821 tablet_bootstrap.cc:654] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.674818 31824 raft_consensus.cc:729] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.675153 31824 consensus_queue.cc:260] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.677009 31827 tablet_bootstrap.cc:492] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:34.677181 31827 ts_tablet_manager.cc:1367] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.005s
I20240503 14:13:34.677945 31827 raft_consensus.cc:348] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.678447 31824 ts_tablet_manager.cc:1398] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 14:13:34.678562 31827 raft_consensus.cc:374] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.678936 31827 raft_consensus.cc:729] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.679184 31827 consensus_queue.cc:260] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.681708 31827 ts_tablet_manager.cc:1398] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:34.682030 31827 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:34.682536 31820 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:34.682756 31820 ts_tablet_manager.cc:1367] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20240503 14:13:34.683526 31820 raft_consensus.cc:348] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.683827 31820 raft_consensus.cc:374] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.683959 31820 raft_consensus.cc:729] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.684309 31820 consensus_queue.cc:260] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.685135 31826 tablet_bootstrap.cc:492] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:34.685312 31826 ts_tablet_manager.cc:1367] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20240503 14:13:34.686544 31826 raft_consensus.cc:348] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.686794 31826 raft_consensus.cc:374] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.686903 31826 raft_consensus.cc:729] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.687171 31826 consensus_queue.cc:260] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.688000 31831 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:34.688179 31820 ts_tablet_manager.cc:1398] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 14:13:34.688199 31831 ts_tablet_manager.cc:1367] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240503 14:13:34.689064 31831 raft_consensus.cc:348] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.689309 31831 raft_consensus.cc:374] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.689455 31831 raft_consensus.cc:729] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.689486 31826 ts_tablet_manager.cc:1398] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:34.689779 31831 consensus_queue.cc:260] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.690261 31827 tablet_bootstrap.cc:654] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.693017 31821 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:34.693265 31821 ts_tablet_manager.cc:1367] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.002s
I20240503 14:13:34.694391 31821 raft_consensus.cc:348] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.694595 31821 raft_consensus.cc:374] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.694713 31831 ts_tablet_manager.cc:1398] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:34.694716 31821 raft_consensus.cc:729] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.695019 31821 consensus_queue.cc:260] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.695973 31821 ts_tablet_manager.cc:1398] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:34.700066 31827 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:34.700232 31827 ts_tablet_manager.cc:1367] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20240503 14:13:34.700951 31827 raft_consensus.cc:348] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.701171 31827 raft_consensus.cc:374] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.701248 31827 raft_consensus.cc:729] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.701468 31827 consensus_queue.cc:260] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.702045 31827 ts_tablet_manager.cc:1398] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:34.742010 31839 raft_consensus.cc:482] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.742190 31839 raft_consensus.cc:504] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.742911 31839 leader_election.cc:288] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:34.743330 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83675aee0d0d4b1794275c4684c30ca0" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:34.743585 30809 raft_consensus.cc:2454] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:34.743700 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83675aee0d0d4b1794275c4684c30ca0" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:34.743930 31072 raft_consensus.cc:2454] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:34.744050 27976 leader_election.cc:302] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:34.744350 31839 raft_consensus.cc:2781] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.744514 31839 raft_consensus.cc:482] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.744612 31839 raft_consensus.cc:3037] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.747014 31839 raft_consensus.cc:504] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.747531 31839 leader_election.cc:288] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:34.747880 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83675aee0d0d4b1794275c4684c30ca0" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:34.747997 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83675aee0d0d4b1794275c4684c30ca0" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:34.748091 30809 raft_consensus.cc:3037] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.748188 31072 raft_consensus.cc:3037] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.750124 30809 raft_consensus.cc:2454] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:34.750280 31072 raft_consensus.cc:2454] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:34.750553 27976 leader_election.cc:302] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:34.750859 31839 raft_consensus.cc:2781] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.751027 31839 raft_consensus.cc:686] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:34.751408 31839 consensus_queue.cc:237] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:34.754427 27894 catalog_manager.cc:5484] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:34.767906 31532 raft_consensus.cc:482] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.768065 31532 raft_consensus.cc:504] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.768575 31532 leader_election.cc:288] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:34.769014 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ba33946424065a465217bae7cdbb0" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:34.769068 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ba33946424065a465217bae7cdbb0" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:34.769207 28173 raft_consensus.cc:2454] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:34.769335 31202 raft_consensus.cc:2454] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:34.769651 31006 leader_election.cc:302] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:34.769968 31532 raft_consensus.cc:2781] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.770083 31532 raft_consensus.cc:482] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.770187 31532 raft_consensus.cc:3037] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.772179 31532 raft_consensus.cc:504] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.772758 31532 leader_election.cc:288] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:34.773129 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ba33946424065a465217bae7cdbb0" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:34.773314 28173 raft_consensus.cc:3037] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.773298 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ba33946424065a465217bae7cdbb0" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:34.773533 31202 raft_consensus.cc:3037] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.775426 28173 raft_consensus.cc:2454] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:34.775897 31006 leader_election.cc:302] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:34.776228 31532 raft_consensus.cc:2781] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.776537 31202 raft_consensus.cc:2454] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:34.777186 31532 raft_consensus.cc:686] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:34.777503 31532 consensus_queue.cc:237] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:34.780589 27894 catalog_manager.cc:5484] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:34.780864 31686 raft_consensus.cc:482] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.781028 31686 raft_consensus.cc:504] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.781626 31686 leader_election.cc:288] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:34.782030 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b82b7a5fb7c4e13b126f6d47164a463" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:34.782269 28040 raft_consensus.cc:2454] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:34.782557 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b82b7a5fb7c4e13b126f6d47164a463" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:34.782742 30877 leader_election.cc:302] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:34.782812 28432 raft_consensus.cc:2454] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:34.782982 31686 raft_consensus.cc:2781] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.783082 31686 raft_consensus.cc:482] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.783161 31686 raft_consensus.cc:3037] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.785277 31686 raft_consensus.cc:504] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.785742 31686 leader_election.cc:288] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:34.786181 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b82b7a5fb7c4e13b126f6d47164a463" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:34.786294 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b82b7a5fb7c4e13b126f6d47164a463" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:34.786399 28040 raft_consensus.cc:3037] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.786509 28432 raft_consensus.cc:3037] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.788375 28040 raft_consensus.cc:2454] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:34.788794 30877 leader_election.cc:302] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:34.788933 28432 raft_consensus.cc:2454] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:34.789206 31762 raft_consensus.cc:2781] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.789366 31762 raft_consensus.cc:686] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:34.789646 31762 consensus_queue.cc:237] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:34.792156 31532 raft_consensus.cc:482] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.792409 31532 raft_consensus.cc:504] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.793463 31532 leader_election.cc:288] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:34.793922 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c283674c342169940a5903516ec93" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:34.793975 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c283674c342169940a5903516ec93" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:34.794200 30940 raft_consensus.cc:2454] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:34.794193 30809 raft_consensus.cc:2454] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:34.794607 31007 leader_election.cc:302] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:34.794924 31532 raft_consensus.cc:2781] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.795099 31532 raft_consensus.cc:482] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.795234 31532 raft_consensus.cc:3037] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.799983 31532 raft_consensus.cc:504] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.800187 27894 catalog_manager.cc:5484] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:34.800940 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c283674c342169940a5903516ec93" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:34.801133 30809 raft_consensus.cc:3037] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.801523 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57c283674c342169940a5903516ec93" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:34.801725 30940 raft_consensus.cc:3037] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.803566 30940 raft_consensus.cc:2454] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:34.803712 30809 raft_consensus.cc:2454] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:34.804131 31532 leader_election.cc:288] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:34.804193 31005 leader_election.cc:302] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:34.804435 31532 raft_consensus.cc:2781] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.804574 31532 raft_consensus.cc:686] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:34.804817 31532 consensus_queue.cc:237] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:34.807878 27894 catalog_manager.cc:5484] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:34.877957 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:34.884066 30809 raft_consensus.cc:1261] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.884063 31072 raft_consensus.cc:1261] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.884722 31839 consensus_queue.cc:1035] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.885212 31839 consensus_queue.cc:1035] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.889256 27894 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 83675aee0d0d4b1794275c4684c30ca0 (table rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670]). Expected version 1 got 0
I20240503 14:13:34.892695 27894 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 83675aee0d0d4b1794275c4684c30ca0 (table rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670]). Expected version 1 got 0
I20240503 14:13:34.894048 31848 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.894847 31848 tablet.cc:1620] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:34.895704 31847 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.896306 31847 tablet.cc:1620] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:34.896791 27894 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 83675aee0d0d4b1794275c4684c30ca0 (table rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670]). Expected version 1 got 0
I20240503 14:13:34.901297 31849 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.901831 31849 tablet.cc:1620] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:34.904598 27896 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 83675aee0d0d4b1794275c4684c30ca0 (table rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670]). Expected version 1 got 0
I20240503 14:13:34.905759 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670] alter complete (version 1)
I20240503 14:13:35.009450 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:35.017529 31847 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.018031 31847 tablet.cc:1620] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:35.018057 31848 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.018772 31849 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.019207 31849 tablet.cc:1620] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:35.022536 31848 tablet.cc:1620] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:35.026728 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=988b7e2894084c3ebe6e550de74bd670] alter complete (version 2)
I20240503 14:13:35.130013 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:35.137776 31847 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:35.138738 31849 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:35.143330 31848 tablet.cc:1722] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:35.148159 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=988b7e2894084c3ebe6e550de74bd670] alter complete (version 3)
I20240503 14:13:35.167876 31762 consensus_queue.cc:1035] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.172760 31762 consensus_queue.cc:1035] T 7b82b7a5fb7c4e13b126f6d47164a463 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.247880 31532 consensus_queue.cc:1035] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.251132 27901 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:35.251374 27901 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:35.252561 31532 consensus_queue.cc:1035] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.254971 27901 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 83675aee0d0d4b1794275c4684c30ca0
I20240503 14:13:35.255826 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 83675aee0d0d4b1794275c4684c30ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:35 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:35.255903 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 83675aee0d0d4b1794275c4684c30ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:35 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:35.256472 31861 tablet_replica.cc:330] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:35.256496 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 83675aee0d0d4b1794275c4684c30ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:35 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:35.256685 31861 raft_consensus.cc:2229] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:35.256899 31861 raft_consensus.cc:2258] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.258051 31861 ts_tablet_manager.cc:1875] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:35.258688 31862 tablet_replica.cc:330] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:35.258965 31862 raft_consensus.cc:2229] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:35.259373 31862 raft_consensus.cc:2258] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.260294 31862 ts_tablet_manager.cc:1875] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:35.262581 31860 tablet_replica.cc:330] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:35.262853 31860 raft_consensus.cc:2229] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:35.263805 31860 raft_consensus.cc:2258] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.265156 31860 ts_tablet_manager.cc:1875] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:35.270430 31860 ts_tablet_manager.cc:1888] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:35.270608 31860 log.cc:1196] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/83675aee0d0d4b1794275c4684c30ca0
I20240503 14:13:35.270681 31862 ts_tablet_manager.cc:1888] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:35.270905 31862 log.cc:1196] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/83675aee0d0d4b1794275c4684c30ca0
I20240503 14:13:35.271131 31860 ts_tablet_manager.cc:1909] T 83675aee0d0d4b1794275c4684c30ca0 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:35.271430 31862 ts_tablet_manager.cc:1909] T 83675aee0d0d4b1794275c4684c30ca0 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:35.272163 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 83675aee0d0d4b1794275c4684c30ca0 (table rebalancer_dynamic_table_new_name [id=988b7e2894084c3ebe6e550de74bd670]) successfully deleted
I20240503 14:13:35.272461 31861 ts_tablet_manager.cc:1888] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:35.272570 31861 log.cc:1196] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/83675aee0d0d4b1794275c4684c30ca0
I20240503 14:13:35.272670 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 83675aee0d0d4b1794275c4684c30ca0 (table rebalancer_dynamic_table_new_name [id=988b7e2894084c3ebe6e550de74bd670]) successfully deleted
I20240503 14:13:35.272965 31861 ts_tablet_manager.cc:1909] T 83675aee0d0d4b1794275c4684c30ca0 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:35.273967 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 83675aee0d0d4b1794275c4684c30ca0 (table rebalancer_dynamic_table_new_name [id=988b7e2894084c3ebe6e550de74bd670]) successfully deleted
I20240503 14:13:35.348512 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:35.358022 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4544aefcc54a828dc3b75726367f70 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.358543 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4544aefcc54a828dc3b75726367f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.360651 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4544aefcc54a828dc3b75726367f70 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.360679 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 36a2e220a0034a9aac672f6ccdf497d1 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.361114 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a2e220a0034a9aac672f6ccdf497d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.361128 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4544aefcc54a828dc3b75726367f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.361682 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 92f507cdcfb04eb4a9eb448773206783 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.362049 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f507cdcfb04eb4a9eb448773206783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.364521 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 92f507cdcfb04eb4a9eb448773206783 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.364893 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f507cdcfb04eb4a9eb448773206783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.366163 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:35.367218 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 92f507cdcfb04eb4a9eb448773206783 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.367664 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f507cdcfb04eb4a9eb448773206783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.369827 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 36a2e220a0034a9aac672f6ccdf497d1 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.370296 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a2e220a0034a9aac672f6ccdf497d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.370571 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 36a2e220a0034a9aac672f6ccdf497d1 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.370970 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a2e220a0034a9aac672f6ccdf497d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.374847 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 1b4544aefcc54a828dc3b75726367f70 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.375061 31532 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.375443 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4544aefcc54a828dc3b75726367f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.381095 31865 tablet_bootstrap.cc:492] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:35.385174 31865 tablet_bootstrap.cc:654] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.392833 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf82d59bb2a48da8a608521c3f1a688 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155]), partition=
I20240503 14:13:35.393237 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf82d59bb2a48da8a608521c3f1a688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.394438 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf82d59bb2a48da8a608521c3f1a688 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155]), partition=
I20240503 14:13:35.394985 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf82d59bb2a48da8a608521c3f1a688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.395377 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf82d59bb2a48da8a608521c3f1a688 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155]), partition=
I20240503 14:13:35.395874 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bf82d59bb2a48da8a608521c3f1a688. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.404994 31532 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.410828 31869 tablet_bootstrap.cc:492] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:35.413127 31869 tablet_bootstrap.cc:654] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.414865 31865 tablet_bootstrap.cc:492] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:35.416128 31865 ts_tablet_manager.cc:1367] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20240503 14:13:35.417532 31865 raft_consensus.cc:348] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.420297 31865 raft_consensus.cc:374] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.420673 31865 raft_consensus.cc:729] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.421324 31865 consensus_queue.cc:260] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.422770 31865 ts_tablet_manager.cc:1398] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:35.426069 31873 tablet_bootstrap.cc:492] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:35.431054 31873 tablet_bootstrap.cc:654] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.431288 31867 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:35.434120 31870 tablet_bootstrap.cc:492] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:35.435886 31870 tablet_bootstrap.cc:654] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.436569 31867 tablet_bootstrap.cc:654] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.446732 31873 tablet_bootstrap.cc:492] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:35.446964 31873 ts_tablet_manager.cc:1367] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.006s
I20240503 14:13:35.448436 31873 raft_consensus.cc:348] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.449524 31870 tablet_bootstrap.cc:492] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:35.450105 31864 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:35.451894 31864 tablet_bootstrap.cc:654] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.452872 31870 ts_tablet_manager.cc:1367] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240503 14:13:35.453514 31865 tablet_bootstrap.cc:492] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:35.455431 31873 raft_consensus.cc:374] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.455552 31873 raft_consensus.cc:729] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.455686 31865 tablet_bootstrap.cc:654] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.455821 31873 consensus_queue.cc:260] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.461571 31873 ts_tablet_manager.cc:1398] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 14:13:35.461726 31870 raft_consensus.cc:348] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.461947 31870 raft_consensus.cc:374] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.462080 31870 raft_consensus.cc:729] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.463526 31873 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:35.465178 31873 tablet_bootstrap.cc:654] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.466153 31870 consensus_queue.cc:260] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.468946 31865 tablet_bootstrap.cc:492] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:35.469157 31865 ts_tablet_manager.cc:1367] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20240503 14:13:35.469187 31869 tablet_bootstrap.cc:492] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:35.469322 31869 ts_tablet_manager.cc:1367] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.026s
I20240503 14:13:35.470006 31865 raft_consensus.cc:348] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.470108 31869 raft_consensus.cc:348] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.470335 31869 raft_consensus.cc:374] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.470379 31865 raft_consensus.cc:374] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.470438 31869 raft_consensus.cc:729] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.470558 31865 raft_consensus.cc:729] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.470671 31869 consensus_queue.cc:260] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.470952 31865 consensus_queue.cc:260] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.471616 31869 ts_tablet_manager.cc:1398] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:35.471881 31869 tablet_bootstrap.cc:492] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:35.473291 31869 tablet_bootstrap.cc:654] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.473512 31865 ts_tablet_manager.cc:1398] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:35.475135 31869 tablet_bootstrap.cc:492] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:35.475334 31869 ts_tablet_manager.cc:1367] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:35.476111 31869 raft_consensus.cc:348] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.476389 31869 raft_consensus.cc:374] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.476517 31869 raft_consensus.cc:729] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.476845 31870 ts_tablet_manager.cc:1398] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240503 14:13:35.476786 31869 consensus_queue.cc:260] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.477234 31870 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:35.478075 31867 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:35.478637 31867 ts_tablet_manager.cc:1367] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.000s
I20240503 14:13:35.479290 31869 ts_tablet_manager.cc:1398] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:35.479483 31870 tablet_bootstrap.cc:654] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.479629 31867 raft_consensus.cc:348] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.479844 31867 raft_consensus.cc:374] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.480003 31867 raft_consensus.cc:729] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.480245 31867 consensus_queue.cc:260] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.481527 31873 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:35.481689 31873 ts_tablet_manager.cc:1367] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20240503 14:13:35.482437 31873 raft_consensus.cc:348] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.482730 31873 raft_consensus.cc:374] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.482806 31867 ts_tablet_manager.cc:1398] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:35.482856 31873 raft_consensus.cc:729] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.483191 31870 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:35.483156 31873 consensus_queue.cc:260] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.483383 31870 ts_tablet_manager.cc:1367] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:35.483449 31864 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:35.483798 31873 ts_tablet_manager.cc:1398] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:35.484388 31870 raft_consensus.cc:348] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.484827 31870 raft_consensus.cc:374] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.484966 31870 raft_consensus.cc:729] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.485244 31812 raft_consensus.cc:482] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.485395 31812 raft_consensus.cc:504] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.485625 31870 consensus_queue.cc:260] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.486282 31812 leader_election.cc:288] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:35.486430 31870 ts_tablet_manager.cc:1398] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:35.486758 31870 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:35.487571 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:35.488572 31870 tablet_bootstrap.cc:654] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:35.488606 28369 leader_election.cc:341] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:35.494251 31864 ts_tablet_manager.cc:1367] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20240503 14:13:35.495651 31864 raft_consensus.cc:348] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.496171 31864 raft_consensus.cc:374] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.496335 31864 raft_consensus.cc:729] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.496706 31864 consensus_queue.cc:260] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.498302 31864 ts_tablet_manager.cc:1398] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:35.500175 31864 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:35.507802 31864 tablet_bootstrap.cc:654] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.511504 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
W20240503 14:13:35.513007 28370 leader_election.cc:341] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:35.513149 28370 leader_election.cc:302] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:13:35.513401 31885 raft_consensus.cc:2726] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:35.517263 31864 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:35.517495 31864 ts_tablet_manager.cc:1367] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20240503 14:13:35.518330 31864 raft_consensus.cc:348] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.518658 31864 raft_consensus.cc:374] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.518764 31864 raft_consensus.cc:729] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.519294 31864 consensus_queue.cc:260] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.520104 31864 ts_tablet_manager.cc:1398] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:35.524426 31870 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:35.524574 31870 ts_tablet_manager.cc:1367] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.038s user 0.016s sys 0.016s
I20240503 14:13:35.525323 31870 raft_consensus.cc:348] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.525517 31870 raft_consensus.cc:374] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.525616 31870 raft_consensus.cc:729] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.525823 31870 consensus_queue.cc:260] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.526378 31870 ts_tablet_manager.cc:1398] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:35.527717 31805 raft_consensus.cc:482] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.527849 31805 raft_consensus.cc:504] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.528613 31805 leader_election.cc:288] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:35.536245 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f507cdcfb04eb4a9eb448773206783" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:35.536530 31072 raft_consensus.cc:2454] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:35.537009 28239 leader_election.cc:302] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:35.537357 31805 raft_consensus.cc:2781] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.537520 31805 raft_consensus.cc:482] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.537617 31805 raft_consensus.cc:3037] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.541558 31805 raft_consensus.cc:504] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.542268 31805 leader_election.cc:288] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:35.543152 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f507cdcfb04eb4a9eb448773206783" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:35.543365 31072 raft_consensus.cc:3037] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.544083 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f507cdcfb04eb4a9eb448773206783" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:35.544312 30940 raft_consensus.cc:2454] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:35.544764 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f507cdcfb04eb4a9eb448773206783" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:35.544997 30941 raft_consensus.cc:3037] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.545902 31072 raft_consensus.cc:2454] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:35.546279 28239 leader_election.cc:302] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:35.546584 31805 raft_consensus.cc:2781] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.546746 31805 raft_consensus.cc:686] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:35.547006 31805 consensus_queue.cc:237] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:35.547191 30941 raft_consensus.cc:2454] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:35.550061 27899 catalog_manager.cc:5484] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:35.611717 31851 raft_consensus.cc:482] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.611927 31851 raft_consensus.cc:504] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.612702 31851 leader_election.cc:288] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:35.613191 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2e220a0034a9aac672f6ccdf497d1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:35.613350 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2e220a0034a9aac672f6ccdf497d1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:35.613487 31072 raft_consensus.cc:2454] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:35.613590 30809 raft_consensus.cc:2454] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:35.614014 27978 leader_election.cc:302] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:35.614396 31851 raft_consensus.cc:2781] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.614529 31851 raft_consensus.cc:482] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.614619 31851 raft_consensus.cc:3037] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.617501 31851 raft_consensus.cc:504] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.618570 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2e220a0034a9aac672f6ccdf497d1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:35.618752 30809 raft_consensus.cc:3037] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.618845 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2e220a0034a9aac672f6ccdf497d1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:35.619052 31072 raft_consensus.cc:3037] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.620790 30809 raft_consensus.cc:2454] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:35.618113 31851 leader_election.cc:288] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:35.621258 27976 leader_election.cc:302] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:35.621507 31851 raft_consensus.cc:2781] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.621635 31851 raft_consensus.cc:686] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:35.621903 31851 consensus_queue.cc:237] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:35.621951 31072 raft_consensus.cc:2454] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:35.628149 27899 catalog_manager.cc:5484] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:35.639833 31762 raft_consensus.cc:482] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.640026 31762 raft_consensus.cc:504] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.640838 31762 leader_election.cc:288] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:35.641232 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf82d59bb2a48da8a608521c3f1a688" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:35.641244 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf82d59bb2a48da8a608521c3f1a688" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:35.641485 28040 raft_consensus.cc:2454] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:35.641479 30809 raft_consensus.cc:2454] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:35.641908 30877 leader_election.cc:302] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:35.642292 31762 raft_consensus.cc:2781] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.642418 31762 raft_consensus.cc:482] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.642508 31762 raft_consensus.cc:3037] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.644518 31762 raft_consensus.cc:504] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.645180 31762 leader_election.cc:288] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:35.645597 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf82d59bb2a48da8a608521c3f1a688" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:35.645812 30809 raft_consensus.cc:3037] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.645824 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf82d59bb2a48da8a608521c3f1a688" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:35.645995 28040 raft_consensus.cc:3037] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.648613 30809 raft_consensus.cc:2454] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:35.648722 28040 raft_consensus.cc:2454] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:35.649099 30874 leader_election.cc:302] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:35.649406 31762 raft_consensus.cc:2781] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.649618 31762 raft_consensus.cc:686] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:35.650183 31762 consensus_queue.cc:237] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:35.653101 27899 catalog_manager.cc:5484] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:35.699110 31762 raft_consensus.cc:482] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.699368 31762 raft_consensus.cc:504] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.700078 31762 leader_election.cc:288] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:35.701267 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:35.701445 28432 raft_consensus.cc:2454] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:35.701884 30877 leader_election.cc:302] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:35.702227 31762 raft_consensus.cc:2781] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.702350 31762 raft_consensus.cc:482] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.702461 31762 raft_consensus.cc:3037] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.704788 31762 raft_consensus.cc:504] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.705421 31762 leader_election.cc:288] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:35.705869 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:35.706082 28432 raft_consensus.cc:3037] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.707141 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:35.707167 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:35.707382 28304 raft_consensus.cc:2454] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:35.708205 28432 raft_consensus.cc:2454] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:35.708580 30877 leader_election.cc:302] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:13:35.708886 31762 raft_consensus.cc:2781] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.709056 31762 raft_consensus.cc:686] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:35.709290 31762 consensus_queue.cc:237] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:35.712335 27898 catalog_manager.cc:5484] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:35.774729 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:35.780447 30809 raft_consensus.cc:1261] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:35.780483 28040 raft_consensus.cc:1261] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:35.780959 31762 consensus_queue.cc:1035] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.781213 31804 consensus_queue.cc:1035] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.785697 27899 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 3bf82d59bb2a48da8a608521c3f1a688 (table rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155]). Expected version 1 got 0
I20240503 14:13:35.785890 31900 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:35.786646 31900 tablet.cc:1620] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:35.787726 31901 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:35.789598 27899 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 3bf82d59bb2a48da8a608521c3f1a688 (table rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155]). Expected version 1 got 0
I20240503 14:13:35.791587 31901 tablet.cc:1620] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:35.798681 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155] alter complete (version 1)
I20240503 14:13:35.799852 27900 catalog_manager.cc:5576] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293) does not have the latest schema for tablet 3bf82d59bb2a48da8a608521c3f1a688 (table rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155]). Expected version 1 got 0
I20240503 14:13:35.799847 31903 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:35.800380 31903 tablet.cc:1620] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:35.903946 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:35.910890 31900 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.911430 31900 tablet.cc:1620] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:35.911757 31903 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.911965 31901 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.912249 31903 tablet.cc:1620] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: MemRowSet was empty: no flush needed.
I20240503 14:13:35.912412 31901 tablet.cc:1620] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:35.921033 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=670e1e9aa9d44f49a76e96c0d0f15155] alter complete (version 2)
I20240503 14:13:35.987557 31805 consensus_queue.cc:1035] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.994163 31805 consensus_queue.cc:1035] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.024713 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:36.031821 31900 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.035480 31903 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.036665 31901 tablet.cc:1722] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.043758 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=670e1e9aa9d44f49a76e96c0d0f15155] alter complete (version 3)
I20240503 14:13:36.097934 31852 consensus_queue.cc:1035] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.101883 31852 consensus_queue.cc:1035] T 36a2e220a0034a9aac672f6ccdf497d1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.147935 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:36.148185 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:36.150972 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 3bf82d59bb2a48da8a608521c3f1a688
I20240503 14:13:36.151849 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bf82d59bb2a48da8a608521c3f1a688 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:36.152273 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bf82d59bb2a48da8a608521c3f1a688 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:52616
I20240503 14:13:36.152987 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bf82d59bb2a48da8a608521c3f1a688 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:36.153980 31912 tablet_replica.cc:330] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:36.154353 31912 raft_consensus.cc:2229] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.154690 31912 raft_consensus.cc:2258] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.155798 31912 ts_tablet_manager.cc:1875] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.163646 31913 tablet_replica.cc:330] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:13:36.164000 31913 raft_consensus.cc:2229] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.164264 31913 raft_consensus.cc:2258] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.164913 31912 ts_tablet_manager.cc:1888] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.165100 31912 log.cc:1196] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/3bf82d59bb2a48da8a608521c3f1a688
I20240503 14:13:36.165270 31913 ts_tablet_manager.cc:1875] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.165899 31912 ts_tablet_manager.cc:1909] T 3bf82d59bb2a48da8a608521c3f1a688 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:36.167296 31914 tablet_replica.cc:330] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:36.168331 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 3bf82d59bb2a48da8a608521c3f1a688 (table rebalancer_dynamic_table_new_name [id=670e1e9aa9d44f49a76e96c0d0f15155]) successfully deleted
I20240503 14:13:36.168823 31914 raft_consensus.cc:2229] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:36.169309 31914 raft_consensus.cc:2258] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.169289 31804 consensus_queue.cc:1035] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.170311 31914 ts_tablet_manager.cc:1875] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.171085 31913 ts_tablet_manager.cc:1888] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.171192 31913 log.cc:1196] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/3bf82d59bb2a48da8a608521c3f1a688
I20240503 14:13:36.171635 31913 ts_tablet_manager.cc:1909] T 3bf82d59bb2a48da8a608521c3f1a688 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting consensus metadata
I20240503 14:13:36.172600 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 3bf82d59bb2a48da8a608521c3f1a688 (table rebalancer_dynamic_table_new_name [id=670e1e9aa9d44f49a76e96c0d0f15155]) successfully deleted
I20240503 14:13:36.175678 28304 raft_consensus.cc:3037] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.179257 31804 consensus_queue.cc:1035] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.180013 31914 ts_tablet_manager.cc:1888] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.180162 31914 log.cc:1196] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/3bf82d59bb2a48da8a608521c3f1a688
I20240503 14:13:36.180620 31914 ts_tablet_manager.cc:1909] T 3bf82d59bb2a48da8a608521c3f1a688 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:36.181798 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 3bf82d59bb2a48da8a608521c3f1a688 (table rebalancer_dynamic_table_new_name [id=670e1e9aa9d44f49a76e96c0d0f15155]) successfully deleted
I20240503 14:13:36.231981 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:36.240543 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 3905bce9ed6345f9979bda9500d1d053 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.240777 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 3905bce9ed6345f9979bda9500d1d053 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.241053 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3905bce9ed6345f9979bda9500d1d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.241215 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3905bce9ed6345f9979bda9500d1d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.241376 31182 tablet_service.cc:1467] Processing CreateTablet for tablet bde4e7b3ac8e47daba3dc5090e943aa7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.241753 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4e7b3ac8e47daba3dc5090e943aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.246312 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 3905bce9ed6345f9979bda9500d1d053 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.246821 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3905bce9ed6345f9979bda9500d1d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.247959 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 7f807f1ac8d8417aaffd9254993b2085 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.248420 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f807f1ac8d8417aaffd9254993b2085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.250638 31051 tablet_service.cc:1467] Processing CreateTablet for tablet bde4e7b3ac8e47daba3dc5090e943aa7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.251128 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4e7b3ac8e47daba3dc5090e943aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.255468 30789 tablet_service.cc:1467] Processing CreateTablet for tablet bde4e7b3ac8e47daba3dc5090e943aa7 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.255542 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:36.255893 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4e7b3ac8e47daba3dc5090e943aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.258729 31918 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:36.259377 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 7f807f1ac8d8417aaffd9254993b2085 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.259793 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f807f1ac8d8417aaffd9254993b2085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.260535 31918 tablet_bootstrap.cc:654] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.262436 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 7f807f1ac8d8417aaffd9254993b2085 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.262611 31918 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:36.262737 31918 ts_tablet_manager.cc:1367] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:36.262847 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f807f1ac8d8417aaffd9254993b2085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.264168 31918 raft_consensus.cc:348] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.264446 31918 raft_consensus.cc:374] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.264536 31918 raft_consensus.cc:729] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.264763 31918 consensus_queue.cc:260] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.275229 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 894944de87c1435bb197f963174a6992 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193]), partition=
I20240503 14:13:36.275595 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894944de87c1435bb197f963174a6992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.275787 30788 tablet_service.cc:1467] Processing CreateTablet for tablet 894944de87c1435bb197f963174a6992 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193]), partition=
I20240503 14:13:36.276039 31918 ts_tablet_manager.cc:1398] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240503 14:13:36.276082 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894944de87c1435bb197f963174a6992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.276335 31918 tablet_bootstrap.cc:492] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:36.277969 31918 tablet_bootstrap.cc:654] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.279606 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 894944de87c1435bb197f963174a6992 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193]), partition=
I20240503 14:13:36.279925 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894944de87c1435bb197f963174a6992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.287555 31924 tablet_bootstrap.cc:492] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:36.289168 31922 tablet_bootstrap.cc:492] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:36.290799 31919 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:36.291198 31924 tablet_bootstrap.cc:654] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.294302 31922 tablet_bootstrap.cc:654] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.295827 31919 tablet_bootstrap.cc:654] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.302979 31926 tablet_bootstrap.cc:492] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:36.305873 31926 tablet_bootstrap.cc:654] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.307618 31919 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:36.307806 31919 ts_tablet_manager.cc:1367] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20240503 14:13:36.308099 31918 tablet_bootstrap.cc:492] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:36.308321 31918 ts_tablet_manager.cc:1367] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.016s
I20240503 14:13:36.308373 31922 tablet_bootstrap.cc:492] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:36.308579 31922 ts_tablet_manager.cc:1367] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20240503 14:13:36.308609 31919 raft_consensus.cc:348] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.308859 31919 raft_consensus.cc:374] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.308954 31924 tablet_bootstrap.cc:492] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:36.308986 31919 raft_consensus.cc:729] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.309110 31924 ts_tablet_manager.cc:1367] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.009s
I20240503 14:13:36.309358 31922 raft_consensus.cc:348] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.309386 31919 consensus_queue.cc:260] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.309576 31922 raft_consensus.cc:374] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.309697 31922 raft_consensus.cc:729] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.309927 31924 raft_consensus.cc:348] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.309926 31918 raft_consensus.cc:348] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.310135 31924 raft_consensus.cc:374] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.310151 31918 raft_consensus.cc:374] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.310215 31918 raft_consensus.cc:729] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.310231 31924 raft_consensus.cc:729] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.309983 31922 consensus_queue.cc:260] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.310513 31924 consensus_queue.cc:260] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.310588 31918 consensus_queue.cc:260] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.311239 31924 ts_tablet_manager.cc:1398] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:36.311226 31918 ts_tablet_manager.cc:1398] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:36.311511 31924 tablet_bootstrap.cc:492] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:36.311532 31918 tablet_bootstrap.cc:492] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:36.311609 31919 ts_tablet_manager.cc:1398] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:36.313987 31924 tablet_bootstrap.cc:654] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.318523 31918 tablet_bootstrap.cc:654] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.318794 31922 ts_tablet_manager.cc:1398] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:13:36.319583 31924 tablet_bootstrap.cc:492] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:36.319757 31924 ts_tablet_manager.cc:1367] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240503 14:13:36.319787 31926 tablet_bootstrap.cc:492] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:36.319978 31926 ts_tablet_manager.cc:1367] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20240503 14:13:36.320731 31924 raft_consensus.cc:348] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.320997 31924 raft_consensus.cc:374] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.321111 31924 raft_consensus.cc:729] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.321105 31926 raft_consensus.cc:348] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.321450 31926 raft_consensus.cc:374] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.321379 31924 consensus_queue.cc:260] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.321697 31922 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:36.321904 31924 ts_tablet_manager.cc:1398] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:36.322520 31926 raft_consensus.cc:729] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.322734 31924 tablet_bootstrap.cc:492] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:36.322867 31926 consensus_queue.cc:260] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.323352 31918 tablet_bootstrap.cc:492] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:36.323508 31918 ts_tablet_manager.cc:1367] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240503 14:13:36.324334 31924 tablet_bootstrap.cc:654] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.325445 31926 ts_tablet_manager.cc:1398] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:36.325749 31926 tablet_bootstrap.cc:492] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:36.325914 31922 tablet_bootstrap.cc:654] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.327447 31918 raft_consensus.cc:348] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.327706 31918 raft_consensus.cc:374] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.328245 31926 tablet_bootstrap.cc:654] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.329655 31918 raft_consensus.cc:729] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.330035 31918 consensus_queue.cc:260] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.333457 31918 ts_tablet_manager.cc:1398] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:13:36.335347 31926 tablet_bootstrap.cc:492] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:36.335552 31926 ts_tablet_manager.cc:1367] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240503 14:13:36.336336 31926 raft_consensus.cc:348] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.336653 31926 raft_consensus.cc:374] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.336738 31926 raft_consensus.cc:729] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.336997 31926 consensus_queue.cc:260] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.338478 31926 ts_tablet_manager.cc:1398] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:36.344424 31922 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:36.344599 31922 ts_tablet_manager.cc:1367] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240503 14:13:36.344724 31924 tablet_bootstrap.cc:492] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:36.344859 31924 ts_tablet_manager.cc:1367] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.001s
I20240503 14:13:36.345467 31922 raft_consensus.cc:348] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.345686 31922 raft_consensus.cc:374] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.345757 31922 raft_consensus.cc:729] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.345574 31924 raft_consensus.cc:348] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.345854 31924 raft_consensus.cc:374] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.345935 31924 raft_consensus.cc:729] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.346007 31922 consensus_queue.cc:260] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.346161 31924 consensus_queue.cc:260] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.346649 31922 ts_tablet_manager.cc:1398] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:36.346947 31922 tablet_bootstrap.cc:492] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:36.347637 31924 ts_tablet_manager.cc:1398] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:36.348430 31922 tablet_bootstrap.cc:654] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.360107 31922 tablet_bootstrap.cc:492] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:36.360256 31922 ts_tablet_manager.cc:1367] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20240503 14:13:36.360896 31922 raft_consensus.cc:348] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.361109 31922 raft_consensus.cc:374] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.361317 31922 raft_consensus.cc:729] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.361568 31922 consensus_queue.cc:260] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.362375 31922 ts_tablet_manager.cc:1398] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:36.369980 31804 raft_consensus.cc:482] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.370184 31804 raft_consensus.cc:504] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.370921 31804 leader_election.cc:288] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:36.371444 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894944de87c1435bb197f963174a6992" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:36.371555 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894944de87c1435bb197f963174a6992" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:36.371708 30809 raft_consensus.cc:2454] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:36.371774 31072 raft_consensus.cc:2454] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:36.372213 30874 leader_election.cc:302] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:36.372483 31804 raft_consensus.cc:2781] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.372579 31804 raft_consensus.cc:482] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.372692 31804 raft_consensus.cc:3037] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.375159 31804 raft_consensus.cc:504] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.375774 31804 leader_election.cc:288] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:36.378998 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894944de87c1435bb197f963174a6992" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:36.379026 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894944de87c1435bb197f963174a6992" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:36.379194 31072 raft_consensus.cc:3037] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.379204 30809 raft_consensus.cc:3037] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.381870 31072 raft_consensus.cc:2454] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:36.382328 30876 leader_election.cc:302] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:36.382874 30809 raft_consensus.cc:2454] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:36.382917 31904 raft_consensus.cc:2781] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.383036 31904 raft_consensus.cc:686] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:36.383342 31904 consensus_queue.cc:237] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:36.395856 27894 catalog_manager.cc:5484] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:36.427742 31690 raft_consensus.cc:482] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.427937 31690 raft_consensus.cc:504] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.428540 31690 leader_election.cc:288] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:36.429003 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905bce9ed6345f9979bda9500d1d053" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:36.429039 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905bce9ed6345f9979bda9500d1d053" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:36.429216 31203 raft_consensus.cc:2454] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:36.429245 31072 raft_consensus.cc:2454] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:36.429634 28109 leader_election.cc:302] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:36.429913 31690 raft_consensus.cc:2781] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.429991 31690 raft_consensus.cc:482] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.430079 31690 raft_consensus.cc:3037] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.431916 31690 raft_consensus.cc:504] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.432592 31690 leader_election.cc:288] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:36.432914 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905bce9ed6345f9979bda9500d1d053" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:36.433104 31203 raft_consensus.cc:3037] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.433195 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3905bce9ed6345f9979bda9500d1d053" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:36.433389 31072 raft_consensus.cc:3037] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.435086 31203 raft_consensus.cc:2454] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:36.435268 31072 raft_consensus.cc:2454] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:36.435485 28108 leader_election.cc:302] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:36.435740 31690 raft_consensus.cc:2781] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.435914 31690 raft_consensus.cc:686] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:36.436188 31690 consensus_queue.cc:237] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:36.439399 27894 catalog_manager.cc:5484] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:36.448772 31921 raft_consensus.cc:482] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.448937 31921 raft_consensus.cc:504] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.449586 31921 leader_election.cc:288] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:36.450059 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4e7b3ac8e47daba3dc5090e943aa7" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:36.450270 31072 raft_consensus.cc:2454] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:36.450317 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4e7b3ac8e47daba3dc5090e943aa7" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:36.450567 30809 raft_consensus.cc:2454] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:36.450613 31138 leader_election.cc:302] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:36.450909 31921 raft_consensus.cc:2781] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.451042 31921 raft_consensus.cc:482] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.451146 31921 raft_consensus.cc:3037] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.453220 31921 raft_consensus.cc:504] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.453752 31921 leader_election.cc:288] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:36.454049 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4e7b3ac8e47daba3dc5090e943aa7" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:36.454208 31072 raft_consensus.cc:3037] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.454301 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4e7b3ac8e47daba3dc5090e943aa7" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:36.454491 30809 raft_consensus.cc:3037] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.455935 31921 raft_consensus.cc:482] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.456079 31921 raft_consensus.cc:504] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.456169 31072 raft_consensus.cc:2454] T bde4e7b3ac8e47daba3dc5090e943aa7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:36.456625 31138 leader_election.cc:302] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:36.456933 30809 raft_consensus.cc:2454] T bde4e7b3ac8e47daba3dc5090e943aa7 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:36.457265 31921 leader_election.cc:288] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:36.457501 31921 raft_consensus.cc:2781] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.457454 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807f1ac8d8417aaffd9254993b2085" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:36.457751 31921 raft_consensus.cc:686] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:36.457713 30810 raft_consensus.cc:2454] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:36.458153 31136 leader_election.cc:302] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:36.458861 31921 consensus_queue.cc:237] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.460592 31940 raft_consensus.cc:2781] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.460693 31940 raft_consensus.cc:482] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.460813 31940 raft_consensus.cc:3037] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.463078 31940 raft_consensus.cc:504] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.463539 31940 leader_election.cc:288] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:36.464111 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807f1ac8d8417aaffd9254993b2085" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:36.464354 30810 raft_consensus.cc:3037] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.465690 27894 catalog_manager.cc:5484] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:36.468170 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807f1ac8d8417aaffd9254993b2085" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:36.468448 30940 raft_consensus.cc:2454] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:36.468611 30810 raft_consensus.cc:2454] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:36.468550 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f807f1ac8d8417aaffd9254993b2085" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:36.468808 30939 raft_consensus.cc:3037] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.469063 31136 leader_election.cc:302] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:36.469885 31939 raft_consensus.cc:2781] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.470049 31939 raft_consensus.cc:686] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:36.470373 31939 consensus_queue.cc:237] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:36.471218 30939 raft_consensus.cc:2454] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:36.474452 27894 catalog_manager.cc:5484] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:36.511318 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:36.516099 31072 raft_consensus.cc:1261] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:36.516155 30810 raft_consensus.cc:1261] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:36.516660 31762 consensus_queue.cc:1035] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.516934 31904 consensus_queue.cc:1035] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.520843 27894 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 894944de87c1435bb197f963174a6992 (table rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193]). Expected version 1 got 0
I20240503 14:13:36.521219 31900 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:36.521701 31900 tablet.cc:1620] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:36.522881 27899 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 894944de87c1435bb197f963174a6992 (table rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193]). Expected version 1 got 0
I20240503 14:13:36.523867 31945 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:36.524556 31945 tablet.cc:1620] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:36.525763 27899 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 894944de87c1435bb197f963174a6992 (table rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193]). Expected version 1 got 0
I20240503 14:13:36.528066 31901 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:36.528612 31901 tablet.cc:1620] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:36.530066 27900 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 894944de87c1435bb197f963174a6992 (table rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193]). Expected version 1 got 0
I20240503 14:13:36.535018 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193] alter complete (version 1)
I20240503 14:13:36.639787 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:36.648504 31900 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:36.649022 31900 tablet.cc:1620] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:36.649099 31945 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:36.649534 31945 tablet.cc:1620] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:36.651849 31901 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:36.652364 31901 tablet.cc:1620] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:36.662513 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=9a9fb28dc9ea40d4a5278ef888d8c193] alter complete (version 2)
I20240503 14:13:36.767607 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:36.775110 31945 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.775789 31901 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.774874 31900 tablet.cc:1722] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.784432 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=9a9fb28dc9ea40d4a5278ef888d8c193] alter complete (version 3)
I20240503 14:13:36.851766 31690 consensus_queue.cc:1035] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.858945 31690 consensus_queue.cc:1035] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.869128 31939 consensus_queue.cc:1035] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.874311 31921 consensus_queue.cc:1035] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.886683 27898 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:36.886924 27898 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:36.889787 27898 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 894944de87c1435bb197f963174a6992
I20240503 14:13:36.890933 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 894944de87c1435bb197f963174a6992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:36.891409 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet 894944de87c1435bb197f963174a6992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:36.891937 31954 tablet_replica.cc:330] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:36.891922 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 894944de87c1435bb197f963174a6992 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:36.892218 31954 raft_consensus.cc:2229] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:36.892691 31954 raft_consensus.cc:2258] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.893568 31954 ts_tablet_manager.cc:1875] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.900054 31954 ts_tablet_manager.cc:1888] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.900228 31954 log.cc:1196] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/894944de87c1435bb197f963174a6992
I20240503 14:13:36.900774 31954 ts_tablet_manager.cc:1909] T 894944de87c1435bb197f963174a6992 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:36.901798 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 894944de87c1435bb197f963174a6992 (table rebalancer_dynamic_table_new_name [id=9a9fb28dc9ea40d4a5278ef888d8c193]) successfully deleted
I20240503 14:13:36.906607 31953 tablet_replica.cc:330] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:36.906648 31955 tablet_replica.cc:330] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:36.906810 31953 raft_consensus.cc:2229] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.906939 31955 raft_consensus.cc:2229] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.907014 31953 raft_consensus.cc:2258] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.907243 31955 raft_consensus.cc:2258] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.907827 31953 ts_tablet_manager.cc:1875] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.908468 31955 ts_tablet_manager.cc:1875] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.910054 31921 consensus_queue.cc:1035] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.914402 31953 ts_tablet_manager.cc:1888] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.914602 31953 log.cc:1196] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/894944de87c1435bb197f963174a6992
I20240503 14:13:36.915127 31953 ts_tablet_manager.cc:1909] T 894944de87c1435bb197f963174a6992 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:36.916049 31955 ts_tablet_manager.cc:1888] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.916198 31955 log.cc:1196] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/894944de87c1435bb197f963174a6992
I20240503 14:13:36.916680 31955 ts_tablet_manager.cc:1909] T 894944de87c1435bb197f963174a6992 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:36.917073 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 894944de87c1435bb197f963174a6992 (table rebalancer_dynamic_table_new_name [id=9a9fb28dc9ea40d4a5278ef888d8c193]) successfully deleted
I20240503 14:13:36.917733 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 894944de87c1435bb197f963174a6992 (table rebalancer_dynamic_table_new_name [id=9a9fb28dc9ea40d4a5278ef888d8c193]) successfully deleted
I20240503 14:13:36.917860 31940 consensus_queue.cc:1035] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.992157 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:36.997180 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:37.001199 30919 tablet_service.cc:1467] Processing CreateTablet for tablet b327b38ed7ae4735a6d2a4e191e34490 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5095496a3d564d1d833cfd41da2d12c5]), partition=
I20240503 14:13:37.001188 28282 tablet_service.cc:1467] Processing CreateTablet for tablet b327b38ed7ae4735a6d2a4e191e34490 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5095496a3d564d1d833cfd41da2d12c5]), partition=
I20240503 14:13:37.001546 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b327b38ed7ae4735a6d2a4e191e34490. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.001559 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b327b38ed7ae4735a6d2a4e191e34490. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.009130 30790 tablet_service.cc:1467] Processing CreateTablet for tablet b327b38ed7ae4735a6d2a4e191e34490 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5095496a3d564d1d833cfd41da2d12c5]), partition=
I20240503 14:13:37.009539 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b327b38ed7ae4735a6d2a4e191e34490. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.015110 31181 tablet_service.cc:1467] Processing CreateTablet for tablet bca56ccf82a74df7bd8006e31e02bd20 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.015640 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca56ccf82a74df7bd8006e31e02bd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.016278 31959 tablet_bootstrap.cc:492] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:37.017791 31959 tablet_bootstrap.cc:654] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.020346 31959 tablet_bootstrap.cc:492] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:37.020512 31959 ts_tablet_manager.cc:1367] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:37.021252 31959 raft_consensus.cc:348] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.021473 31959 raft_consensus.cc:374] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.021565 31959 raft_consensus.cc:729] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.021809 31959 consensus_queue.cc:260] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.022400 31959 ts_tablet_manager.cc:1398] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:37.024312 31050 tablet_service.cc:1467] Processing CreateTablet for tablet bca56ccf82a74df7bd8006e31e02bd20 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.024765 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca56ccf82a74df7bd8006e31e02bd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.025846 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 06e418c4c2cf46608ba454d48fae9285 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.026228 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e418c4c2cf46608ba454d48fae9285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.029675 30788 tablet_service.cc:1467] Processing CreateTablet for tablet bca56ccf82a74df7bd8006e31e02bd20 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.030122 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bca56ccf82a74df7bd8006e31e02bd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.045028 30788 tablet_service.cc:1467] Processing CreateTablet for tablet 06e418c4c2cf46608ba454d48fae9285 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.045444 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e418c4c2cf46608ba454d48fae9285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.046008 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 130a2a0b5aa344d3be535cfde8990d39 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.046808 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130a2a0b5aa344d3be535cfde8990d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.046854 31964 tablet_bootstrap.cc:492] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:37.050647 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 06e418c4c2cf46608ba454d48fae9285 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.051059 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e418c4c2cf46608ba454d48fae9285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.053983 31964 tablet_bootstrap.cc:654] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.054826 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 130a2a0b5aa344d3be535cfde8990d39 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.055258 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130a2a0b5aa344d3be535cfde8990d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.059363 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 130a2a0b5aa344d3be535cfde8990d39 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.059850 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130a2a0b5aa344d3be535cfde8990d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.063015 31963 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:37.065332 31961 tablet_bootstrap.cc:492] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:37.066484 31965 tablet_bootstrap.cc:492] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:37.067061 31961 tablet_bootstrap.cc:654] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.064816 31962 tablet_bootstrap.cc:492] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:37.069077 31962 tablet_bootstrap.cc:654] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.071997 31963 tablet_bootstrap.cc:654] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.074525 31965 tablet_bootstrap.cc:654] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.077268 31964 tablet_bootstrap.cc:492] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:37.077874 31964 ts_tablet_manager.cc:1367] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.000s
I20240503 14:13:37.079471 31964 raft_consensus.cc:348] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.080842 31964 raft_consensus.cc:374] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.081029 31964 raft_consensus.cc:729] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.081363 31964 consensus_queue.cc:260] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.082089 31964 ts_tablet_manager.cc:1398] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:37.083190 31964 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:37.085686 31963 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:37.085916 31963 ts_tablet_manager.cc:1367] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240503 14:13:37.086877 31963 raft_consensus.cc:348] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.087531 31963 raft_consensus.cc:374] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.087711 31963 raft_consensus.cc:729] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.088006 31963 consensus_queue.cc:260] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.088166 31964 tablet_bootstrap.cc:654] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.090526 31963 ts_tablet_manager.cc:1398] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:37.091835 31963 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:37.094055 31973 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:37.095666 31962 tablet_bootstrap.cc:492] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:37.095816 31962 ts_tablet_manager.cc:1367] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.009s
I20240503 14:13:37.096503 31963 tablet_bootstrap.cc:654] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.096504 31973 tablet_bootstrap.cc:654] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.096403 31962 raft_consensus.cc:348] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.096596 31962 raft_consensus.cc:374] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.097334 31961 tablet_bootstrap.cc:492] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:37.097517 31961 ts_tablet_manager.cc:1367] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.012s
I20240503 14:13:37.098338 31962 raft_consensus.cc:729] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.098279 31961 raft_consensus.cc:348] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.098532 31961 raft_consensus.cc:374] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.098655 31961 raft_consensus.cc:729] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.098661 31965 tablet_bootstrap.cc:492] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:37.098644 31962 consensus_queue.cc:260] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.098942 31965 ts_tablet_manager.cc:1367] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240503 14:13:37.098907 31961 consensus_queue.cc:260] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.099349 31962 ts_tablet_manager.cc:1398] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:37.099622 31961 ts_tablet_manager.cc:1398] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:37.099817 31965 raft_consensus.cc:348] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.100019 31961 tablet_bootstrap.cc:492] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:37.100025 31965 raft_consensus.cc:374] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.100116 31965 raft_consensus.cc:729] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.100324 31965 consensus_queue.cc:260] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.101786 31961 tablet_bootstrap.cc:654] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.103459 31965 ts_tablet_manager.cc:1398] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:37.104148 31961 tablet_bootstrap.cc:492] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:37.104324 31961 ts_tablet_manager.cc:1367] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:37.105067 31961 raft_consensus.cc:348] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.105273 31961 raft_consensus.cc:374] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.105355 31961 raft_consensus.cc:729] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.105576 31961 consensus_queue.cc:260] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.106216 31961 ts_tablet_manager.cc:1398] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:37.106562 31961 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:37.108000 31961 tablet_bootstrap.cc:654] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.114969 31973 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:37.115628 31937 raft_consensus.cc:482] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.115718 31973 ts_tablet_manager.cc:1367] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240503 14:13:37.115748 31937 raft_consensus.cc:504] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.116389 31937 leader_election.cc:288] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:37.116552 31973 raft_consensus.cc:348] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.116990 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca56ccf82a74df7bd8006e31e02bd20" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:37.117169 31071 raft_consensus.cc:2454] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:37.117607 30745 leader_election.cc:302] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:37.117923 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca56ccf82a74df7bd8006e31e02bd20" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:37.118095 31964 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:37.118155 31203 raft_consensus.cc:2454] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:37.118278 31964 ts_tablet_manager.cc:1367] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.009s
I20240503 14:13:37.119041 31937 raft_consensus.cc:2781] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.119163 31937 raft_consensus.cc:482] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.119246 31937 raft_consensus.cc:3037] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.119081 31964 raft_consensus.cc:348] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.119339 31964 raft_consensus.cc:374] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.119413 31964 raft_consensus.cc:729] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.119670 31964 consensus_queue.cc:260] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.121114 31964 ts_tablet_manager.cc:1398] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:37.121378 31973 raft_consensus.cc:374] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.121521 31973 raft_consensus.cc:729] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.121805 31973 consensus_queue.cc:260] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.124065 31973 ts_tablet_manager.cc:1398] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:37.124198 31937 raft_consensus.cc:504] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.125097 31963 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:37.125102 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca56ccf82a74df7bd8006e31e02bd20" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:37.125298 31203 raft_consensus.cc:3037] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.125476 31963 ts_tablet_manager.cc:1367] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240503 14:13:37.126204 31963 raft_consensus.cc:348] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.126488 31963 raft_consensus.cc:374] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.126587 31963 raft_consensus.cc:729] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.126832 31963 consensus_queue.cc:260] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.127609 31963 ts_tablet_manager.cc:1398] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:37.127861 31203 raft_consensus.cc:2454] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:37.128345 30744 leader_election.cc:302] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:37.134061 31852 raft_consensus.cc:482] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.134188 31852 raft_consensus.cc:504] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.134862 31852 leader_election.cc:288] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:37.136562 31961 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:37.136772 31961 ts_tablet_manager.cc:1367] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.015s
I20240503 14:13:37.137691 31961 raft_consensus.cc:348] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.137971 31961 raft_consensus.cc:374] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.138136 31961 raft_consensus.cc:729] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.138458 31961 consensus_queue.cc:260] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.138876 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e418c4c2cf46608ba454d48fae9285" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:37.139137 31203 raft_consensus.cc:2454] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:37.139588 27977 leader_election.cc:302] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:37.140038 31852 raft_consensus.cc:2781] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.140156 31852 raft_consensus.cc:482] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.140244 31852 raft_consensus.cc:3037] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.141564 31978 raft_consensus.cc:2781] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.142277 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e418c4c2cf46608ba454d48fae9285" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
W20240503 14:13:37.143045 27976 leader_election.cc:341] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:37.144183 31852 raft_consensus.cc:504] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.145654 31852 leader_election.cc:288] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:37.145835 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e418c4c2cf46608ba454d48fae9285" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:37.146080 31202 raft_consensus.cc:3037] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.147403 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e418c4c2cf46608ba454d48fae9285" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
W20240503 14:13:37.147889 27976 leader_election.cc:341] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:37.148057 31937 leader_election.cc:288] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:37.150385 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bca56ccf82a74df7bd8006e31e02bd20" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:37.150677 31071 raft_consensus.cc:3037] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.150830 31978 raft_consensus.cc:686] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:37.151275 31978 consensus_queue.cc:237] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.151481 31961 ts_tablet_manager.cc:1398] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20240503 14:13:37.151794 31961 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:37.153474 31071 raft_consensus.cc:2454] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:37.153534 31961 tablet_bootstrap.cc:654] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.154835 27898 catalog_manager.cc:5484] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:37.158202 31885 raft_consensus.cc:482] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.158406 31885 raft_consensus.cc:504] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.159173 31885 leader_election.cc:288] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:37.159606 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130a2a0b5aa344d3be535cfde8990d39" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:37.159850 30810 raft_consensus.cc:2454] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:37.160445 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130a2a0b5aa344d3be535cfde8990d39" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:37.160670 28040 raft_consensus.cc:2454] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:37.161104 31961 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:37.161106 28367 leader_election.cc:302] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:37.161255 31961 ts_tablet_manager.cc:1367] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240503 14:13:37.161475 31911 raft_consensus.cc:2781] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.161563 31911 raft_consensus.cc:482] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.161651 31911 raft_consensus.cc:3037] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.162029 31961 raft_consensus.cc:348] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.162317 31961 raft_consensus.cc:374] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.162417 31961 raft_consensus.cc:729] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.162794 31961 consensus_queue.cc:260] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.163328 31961 ts_tablet_manager.cc:1398] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:37.164211 31911 raft_consensus.cc:504] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.165437 31911 leader_election.cc:288] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:37.165876 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130a2a0b5aa344d3be535cfde8990d39" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:37.166077 30810 raft_consensus.cc:3037] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.166745 31202 raft_consensus.cc:2454] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:37.167835 27977 leader_election.cc:302] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:13:37.168120 31852 raft_consensus.cc:2781] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.168259 31852 raft_consensus.cc:686] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:37.168519 31852 consensus_queue.cc:237] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.168772 30810 raft_consensus.cc:2454] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:37.169183 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "130a2a0b5aa344d3be535cfde8990d39" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:37.169199 28367 leader_election.cc:302] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:37.169373 28042 raft_consensus.cc:3037] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.169525 31911 raft_consensus.cc:2781] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.169883 31911 raft_consensus.cc:686] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:37.172528 28042 raft_consensus.cc:2454] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:37.176175 31911 consensus_queue.cc:237] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.180564 27898 catalog_manager.cc:5484] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:37.181532 27900 catalog_manager.cc:5484] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:37.268774 31978 raft_consensus.cc:482] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.268954 31978 raft_consensus.cc:504] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.269500 31978 leader_election.cc:288] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:37.269905 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327b38ed7ae4735a6d2a4e191e34490" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:37.269943 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327b38ed7ae4735a6d2a4e191e34490" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:37.270169 28303 raft_consensus.cc:2454] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:37.270161 30939 raft_consensus.cc:2454] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:37.270658 30746 leader_election.cc:302] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:37.271019 31978 raft_consensus.cc:2781] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.271128 31978 raft_consensus.cc:482] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.271358 31978 raft_consensus.cc:3037] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.273413 31978 raft_consensus.cc:504] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.274091 31978 leader_election.cc:288] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:37.274471 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327b38ed7ae4735a6d2a4e191e34490" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:37.274700 28303 raft_consensus.cc:3037] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.274909 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b327b38ed7ae4735a6d2a4e191e34490" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:37.275137 30939 raft_consensus.cc:3037] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.276872 28303 raft_consensus.cc:2454] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:37.277119 30939 raft_consensus.cc:2454] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:37.277279 30746 leader_election.cc:302] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:37.277561 31978 raft_consensus.cc:2781] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.277755 31978 raft_consensus.cc:686] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:37.277997 31978 consensus_queue.cc:237] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.281265 27900 catalog_manager.cc:5484] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:37.383106 31978 raft_consensus.cc:482] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.383292 31978 raft_consensus.cc:504] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.383955 31978 leader_election.cc:288] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:37.384514 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e418c4c2cf46608ba454d48fae9285" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:37.384796 31202 raft_consensus.cc:2379] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0dbfb0e43f948c3922926aacc365fc1 in current term 1: Already voted for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in this term.
I20240503 14:13:37.389700 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e418c4c2cf46608ba454d48fae9285" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:37.390271 30746 leader_election.cc:302] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:13:37.390614 31978 raft_consensus.cc:3037] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.392881 31978 raft_consensus.cc:2726] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:37.431675 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:37.438043 30939 raft_consensus.cc:1261] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.438679 28303 raft_consensus.cc:1261] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.438967 31979 consensus_queue.cc:1035] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.439177 31978 consensus_queue.cc:1035] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.444171 27900 catalog_manager.cc:5576] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889) does not have the latest schema for tablet b327b38ed7ae4735a6d2a4e191e34490 (table rebalancer_dynamic_table [id=5095496a3d564d1d833cfd41da2d12c5]). Expected version 1 got 0
I20240503 14:13:37.446141 31995 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:37.446780 31995 tablet.cc:1620] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:37.454548 31994 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:37.455188 31994 tablet.cc:1620] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:37.461221 27898 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet b327b38ed7ae4735a6d2a4e191e34490 (table rebalancer_dynamic_table [id=5095496a3d564d1d833cfd41da2d12c5]). Expected version 1 got 0
I20240503 14:13:37.461850 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=5095496a3d564d1d833cfd41da2d12c5] alter complete (version 1)
I20240503 14:13:37.462555 31996 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:37.464390 31996 tablet.cc:1620] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:37.566048 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:37.573251 31994 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:37.573835 31994 tablet.cc:1620] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:37.574398 31995 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:37.574638 31996 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:37.575104 31996 tablet.cc:1620] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:37.575389 31995 tablet.cc:1620] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:37.582753 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=5095496a3d564d1d833cfd41da2d12c5] alter complete (version 2)
I20240503 14:13:37.589320 31852 consensus_queue.cc:1035] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.594252 31852 consensus_queue.cc:1035] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.615975 31911 consensus_queue.cc:1035] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.636616 31885 consensus_queue.cc:1035] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.686551 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:37.693794 31994 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:37.694839 31996 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:37.697440 31995 tablet.cc:1722] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:37.701987 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=5095496a3d564d1d833cfd41da2d12c5] alter complete (version 3)
I20240503 14:13:37.713871 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:37.721777 28282 tablet_service.cc:1467] Processing CreateTablet for tablet c9048a340c8c49f3be2c372aac871118 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.721815 31050 tablet_service.cc:1467] Processing CreateTablet for tablet c9048a340c8c49f3be2c372aac871118 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.722209 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9048a340c8c49f3be2c372aac871118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.722172 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf751c4f28d4358b4f47c3984cfd376 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.722558 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf751c4f28d4358b4f47c3984cfd376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.723109 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 1af2fb3fb3fd4dfe873d124232d812b5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.723507 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af2fb3fb3fd4dfe873d124232d812b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.723479 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf751c4f28d4358b4f47c3984cfd376 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.723846 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf751c4f28d4358b4f47c3984cfd376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.727519 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 3cf751c4f28d4358b4f47c3984cfd376 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.727924 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cf751c4f28d4358b4f47c3984cfd376. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.729089 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 1af2fb3fb3fd4dfe873d124232d812b5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.729447 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af2fb3fb3fd4dfe873d124232d812b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.722231 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9048a340c8c49f3be2c372aac871118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.731571 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 1af2fb3fb3fd4dfe873d124232d812b5 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.731976 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af2fb3fb3fd4dfe873d124232d812b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.735378 30790 tablet_service.cc:1467] Processing CreateTablet for tablet c9048a340c8c49f3be2c372aac871118 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:37.735838 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9048a340c8c49f3be2c372aac871118. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.745535 32008 tablet_bootstrap.cc:492] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:37.746294 31978 consensus_queue.cc:1035] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.750103 32008 tablet_bootstrap.cc:654] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.756305 32010 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:37.757932 32010 tablet_bootstrap.cc:654] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.759296 32013 tablet_bootstrap.cc:492] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:37.760551 32007 tablet_bootstrap.cc:492] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:37.761118 32013 tablet_bootstrap.cc:654] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.762240 32008 tablet_bootstrap.cc:492] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:37.762460 32008 ts_tablet_manager.cc:1367] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 14:13:37.763337 32007 tablet_bootstrap.cc:654] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.763276 32008 raft_consensus.cc:348] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.763813 32008 raft_consensus.cc:374] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.763957 32008 raft_consensus.cc:729] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.764242 32012 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:37.764257 32008 consensus_queue.cc:260] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.765942 31980 consensus_queue.cc:1035] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.766544 32012 tablet_bootstrap.cc:654] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.767927 32006 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:37.769004 32008 ts_tablet_manager.cc:1398] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:37.770787 32006 tablet_bootstrap.cc:654] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.776702 32006 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:37.777359 32006 ts_tablet_manager.cc:1367] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 14:13:37.778483 32006 raft_consensus.cc:348] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.780223 32006 raft_consensus.cc:374] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.780496 32006 raft_consensus.cc:729] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.780895 32006 consensus_queue.cc:260] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.781386 32010 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:37.781522 32007 tablet_bootstrap.cc:492] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:37.781563 32010 ts_tablet_manager.cc:1367] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20240503 14:13:37.782032 32007 ts_tablet_manager.cc:1367] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240503 14:13:37.782085 32006 ts_tablet_manager.cc:1398] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:37.783046 32007 raft_consensus.cc:348] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.783721 32010 raft_consensus.cc:348] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.784554 32010 raft_consensus.cc:374] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.784633 32007 raft_consensus.cc:374] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.784706 32010 raft_consensus.cc:729] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.784971 32007 raft_consensus.cc:729] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.784968 32010 consensus_queue.cc:260] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.785583 32007 consensus_queue.cc:260] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.785944 32012 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:37.786137 32012 ts_tablet_manager.cc:1367] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.006s
I20240503 14:13:37.786743 32007 ts_tablet_manager.cc:1398] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:37.787535 32007 tablet_bootstrap.cc:492] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:37.788022 32012 raft_consensus.cc:348] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.788304 32012 raft_consensus.cc:374] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.788441 32012 raft_consensus.cc:729] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.789053 32010 ts_tablet_manager.cc:1398] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:13:37.789400 32007 tablet_bootstrap.cc:654] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.791146 32013 tablet_bootstrap.cc:492] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:37.791512 32012 consensus_queue.cc:260] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.792845 32013 ts_tablet_manager.cc:1367] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240503 14:13:37.793759 32013 raft_consensus.cc:348] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.794843 32012 ts_tablet_manager.cc:1398] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:13:37.795167 32012 tablet_bootstrap.cc:492] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:37.794073 32013 raft_consensus.cc:374] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.795390 32013 raft_consensus.cc:729] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.795680 32013 consensus_queue.cc:260] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.796345 32013 ts_tablet_manager.cc:1398] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:37.797446 32010 tablet_bootstrap.cc:492] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:37.799044 32010 tablet_bootstrap.cc:654] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.801649 32007 tablet_bootstrap.cc:492] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:37.801863 32007 ts_tablet_manager.cc:1367] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 14:13:37.802724 32007 raft_consensus.cc:348] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.803027 32007 raft_consensus.cc:374] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.803151 32007 raft_consensus.cc:729] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.803458 32007 consensus_queue.cc:260] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.804078 32007 ts_tablet_manager.cc:1398] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:37.805740 32012 tablet_bootstrap.cc:654] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.806087 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:37.806439 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:37.807931 32010 tablet_bootstrap.cc:492] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:37.808087 32010 ts_tablet_manager.cc:1367] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240503 14:13:37.808988 32010 raft_consensus.cc:348] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.809228 32010 raft_consensus.cc:374] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.809336 32010 raft_consensus.cc:729] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.809583 32010 consensus_queue.cc:260] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.810282 32010 ts_tablet_manager.cc:1398] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:13:37.810575 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet b327b38ed7ae4735a6d2a4e191e34490
I20240503 14:13:37.811511 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet b327b38ed7ae4735a6d2a4e191e34490 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:37 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:37.811621 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet b327b38ed7ae4735a6d2a4e191e34490 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:37 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:37.812278 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet b327b38ed7ae4735a6d2a4e191e34490 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:37 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:37.812358 32024 tablet_replica.cc:330] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:37.812623 32024 raft_consensus.cc:2229] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:37.812918 32024 raft_consensus.cc:2258] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:37.813397 32025 tablet_replica.cc:330] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:37.813654 32012 tablet_bootstrap.cc:492] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:37.814314 32024 ts_tablet_manager.cc:1875] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:37.814363 32012 ts_tablet_manager.cc:1367] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.001s
I20240503 14:13:37.814462 32025 raft_consensus.cc:2229] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:37.814970 32025 raft_consensus.cc:2258] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:37.815109 32012 raft_consensus.cc:348] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.815333 32012 raft_consensus.cc:374] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.815404 32012 raft_consensus.cc:729] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.815652 32012 consensus_queue.cc:260] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:37.816207 32012 ts_tablet_manager.cc:1398] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:37.816465 32026 tablet_replica.cc:330] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:37.816637 32026 raft_consensus.cc:2229] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:37.816846 32025 ts_tablet_manager.cc:1875] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:37.816967 32026 raft_consensus.cc:2258] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:37.817867 32026 ts_tablet_manager.cc:1875] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:37.821560 32024 ts_tablet_manager.cc:1888] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:37.821673 32024 log.cc:1196] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/b327b38ed7ae4735a6d2a4e191e34490
I20240503 14:13:37.822113 32024 ts_tablet_manager.cc:1909] T b327b38ed7ae4735a6d2a4e191e34490 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:37.822958 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet b327b38ed7ae4735a6d2a4e191e34490 (table rebalancer_dynamic_table_new_name [id=5095496a3d564d1d833cfd41da2d12c5]) successfully deleted
I20240503 14:13:37.824219 32026 ts_tablet_manager.cc:1888] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:37.824335 32026 log.cc:1196] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/b327b38ed7ae4735a6d2a4e191e34490
I20240503 14:13:37.824748 32025 ts_tablet_manager.cc:1888] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:37.824782 32026 ts_tablet_manager.cc:1909] T b327b38ed7ae4735a6d2a4e191e34490 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:37.824882 32025 log.cc:1196] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/b327b38ed7ae4735a6d2a4e191e34490
I20240503 14:13:37.825310 32025 ts_tablet_manager.cc:1909] T b327b38ed7ae4735a6d2a4e191e34490 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:37.825804 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet b327b38ed7ae4735a6d2a4e191e34490 (table rebalancer_dynamic_table_new_name [id=5095496a3d564d1d833cfd41da2d12c5]) successfully deleted
I20240503 14:13:37.826179 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet b327b38ed7ae4735a6d2a4e191e34490 (table rebalancer_dynamic_table_new_name [id=5095496a3d564d1d833cfd41da2d12c5]) successfully deleted
I20240503 14:13:37.829658 31841 raft_consensus.cc:482] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.829806 31841 raft_consensus.cc:504] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.830578 31841 leader_election.cc:288] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:37.831071 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9048a340c8c49f3be2c372aac871118" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:37.831305 28303 raft_consensus.cc:2454] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:37.831423 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9048a340c8c49f3be2c372aac871118" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:37.831621 30810 raft_consensus.cc:2454] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:37.831689 31008 leader_election.cc:302] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:37.831934 31841 raft_consensus.cc:2781] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.832062 31841 raft_consensus.cc:482] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.832165 31841 raft_consensus.cc:3037] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.834000 31841 raft_consensus.cc:504] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.834527 31841 leader_election.cc:288] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:37.834910 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9048a340c8c49f3be2c372aac871118" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:37.834920 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9048a340c8c49f3be2c372aac871118" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:37.835069 28303 raft_consensus.cc:3037] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.835086 30810 raft_consensus.cc:3037] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.837188 28303 raft_consensus.cc:2454] T c9048a340c8c49f3be2c372aac871118 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:37.837194 30810 raft_consensus.cc:2454] T c9048a340c8c49f3be2c372aac871118 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:37.837538 31005 leader_election.cc:302] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:37.837807 31841 raft_consensus.cc:2781] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.837977 31841 raft_consensus.cc:686] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:37.838214 31841 consensus_queue.cc:237] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:37.841179 27898 catalog_manager.cc:5484] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:37.911327 31841 raft_consensus.cc:482] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.911535 31841 raft_consensus.cc:504] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.912175 31841 leader_election.cc:288] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:37.912611 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:37.912705 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:37.912874 28173 raft_consensus.cc:2454] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:37.912930 31203 raft_consensus.cc:2454] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:37.913345 31006 leader_election.cc:302] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:37.913625 31841 raft_consensus.cc:2781] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.913715 31841 raft_consensus.cc:482] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.913822 31841 raft_consensus.cc:3037] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.914494 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:37.916347 31841 raft_consensus.cc:504] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.916868 31841 leader_election.cc:288] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:37.917384 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:37.917507 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:37.917584 31203 raft_consensus.cc:3037] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.917718 28173 raft_consensus.cc:3037] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.919710 31203 raft_consensus.cc:2454] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:37.920128 31006 leader_election.cc:302] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:37.920270 28173 raft_consensus.cc:2454] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:37.920846 31841 raft_consensus.cc:2781] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.921042 31841 raft_consensus.cc:686] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:37.921293 31841 consensus_queue.cc:237] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:37.923143 31050 tablet_service.cc:1467] Processing CreateTablet for tablet a93d068daa0e466581ce2e44a86a055a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f]), partition=
I20240503 14:13:37.923799 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93d068daa0e466581ce2e44a86a055a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.923719 30919 tablet_service.cc:1467] Processing CreateTablet for tablet a93d068daa0e466581ce2e44a86a055a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f]), partition=
I20240503 14:13:37.924459 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93d068daa0e466581ce2e44a86a055a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.926715 30790 tablet_service.cc:1467] Processing CreateTablet for tablet a93d068daa0e466581ce2e44a86a055a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f]), partition=
I20240503 14:13:37.927472 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93d068daa0e466581ce2e44a86a055a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.928439 27898 catalog_manager.cc:5484] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:37.934013 32008 tablet_bootstrap.cc:492] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:37.934597 32010 tablet_bootstrap.cc:492] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:37.934770 32013 tablet_bootstrap.cc:492] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:37.935603 32008 tablet_bootstrap.cc:654] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.936085 32010 tablet_bootstrap.cc:654] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.936131 32013 tablet_bootstrap.cc:654] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.937928 32008 tablet_bootstrap.cc:492] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:37.938128 32008 ts_tablet_manager.cc:1367] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:37.940340 32008 raft_consensus.cc:348] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.940654 32008 raft_consensus.cc:374] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.940965 32008 raft_consensus.cc:729] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.941329 32008 consensus_queue.cc:260] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.942274 32008 ts_tablet_manager.cc:1398] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:37.943724 32013 tablet_bootstrap.cc:492] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:37.943938 32013 ts_tablet_manager.cc:1367] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 14:13:37.944924 32013 raft_consensus.cc:348] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.945210 32013 raft_consensus.cc:374] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.945348 32013 raft_consensus.cc:729] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.945669 32013 consensus_queue.cc:260] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.947650 32013 ts_tablet_manager.cc:1398] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:37.948431 32010 tablet_bootstrap.cc:492] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:37.948594 32010 ts_tablet_manager.cc:1367] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240503 14:13:37.949355 32010 raft_consensus.cc:348] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.949575 32010 raft_consensus.cc:374] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.949702 32010 raft_consensus.cc:729] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.950085 32010 consensus_queue.cc:260] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:37.950629 32010 ts_tablet_manager.cc:1398] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:38.027695 32005 raft_consensus.cc:482] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.027879 32005 raft_consensus.cc:504] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:38.028595 32005 leader_election.cc:288] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:38.028970 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d068daa0e466581ce2e44a86a055a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:38.029076 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d068daa0e466581ce2e44a86a055a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:38.029186 31071 raft_consensus.cc:2454] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:38.029322 30810 raft_consensus.cc:2454] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:38.029539 30876 leader_election.cc:302] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:38.029763 32005 raft_consensus.cc:2781] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.029897 32005 raft_consensus.cc:482] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.029991 32005 raft_consensus.cc:3037] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.032127 32005 raft_consensus.cc:504] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:38.032610 32005 leader_election.cc:288] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:38.032990 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d068daa0e466581ce2e44a86a055a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:38.033062 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a93d068daa0e466581ce2e44a86a055a" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:38.033175 31071 raft_consensus.cc:3037] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.033259 30810 raft_consensus.cc:3037] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.035202 31071 raft_consensus.cc:2454] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:38.036015 30810 raft_consensus.cc:2454] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:38.035658 30876 leader_election.cc:302] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:38.038894 32005 raft_consensus.cc:2781] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.039211 32005 raft_consensus.cc:686] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:38.039512 32005 consensus_queue.cc:237] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:38.042882 27899 catalog_manager.cc:5484] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:38.156046 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:38.161912 30810 raft_consensus.cc:1261] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.161926 31071 raft_consensus.cc:1261] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.162415 32005 consensus_queue.cc:1035] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.162680 31762 consensus_queue.cc:1035] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.171741 27899 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet a93d068daa0e466581ce2e44a86a055a (table rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f]). Expected version 1 got 0
I20240503 14:13:38.176743 31996 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:38.177299 31996 tablet.cc:1620] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:38.183495 32040 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:38.184023 31994 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:38.184067 27899 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet a93d068daa0e466581ce2e44a86a055a (table rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f]). Expected version 1 got 0
I20240503 14:13:38.184468 31994 tablet.cc:1620] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:38.185518 32040 tablet.cc:1620] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:38.191282 31762 raft_consensus.cc:482] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.191464 31762 raft_consensus.cc:504] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.192057 31762 leader_election.cc:288] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:38.192492 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf751c4f28d4358b4f47c3984cfd376" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:38.191900 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f] alter complete (version 1)
I20240503 14:13:38.192746 28303 raft_consensus.cc:2454] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:38.194274 30877 leader_election.cc:302] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:38.194916 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf751c4f28d4358b4f47c3984cfd376" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:38.195166 31202 raft_consensus.cc:2454] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:38.195281 32005 raft_consensus.cc:2781] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.195367 32005 raft_consensus.cc:482] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.195461 32005 raft_consensus.cc:3037] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.198004 32005 raft_consensus.cc:504] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.198676 32005 leader_election.cc:288] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:38.199110 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf751c4f28d4358b4f47c3984cfd376" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:38.199335 28303 raft_consensus.cc:3037] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.199733 27899 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet a93d068daa0e466581ce2e44a86a055a (table rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f]). Expected version 1 got 0
I20240503 14:13:38.201421 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cf751c4f28d4358b4f47c3984cfd376" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:38.201640 31202 raft_consensus.cc:3037] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.202100 28303 raft_consensus.cc:2454] T 3cf751c4f28d4358b4f47c3984cfd376 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:38.202500 30877 leader_election.cc:302] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:38.202802 32005 raft_consensus.cc:2781] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.203243 32005 raft_consensus.cc:686] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:38.203879 32005 consensus_queue.cc:237] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.204289 31202 raft_consensus.cc:2454] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:38.207187 27898 catalog_manager.cc:5484] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:38.259423 31841 consensus_queue.cc:1035] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.263495 31967 consensus_queue.cc:1035] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.297622 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:38.304767 31996 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:38.305411 31996 tablet.cc:1620] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:38.306466 31994 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:38.306780 32040 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:38.306915 31994 tablet.cc:1620] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:38.307282 32040 tablet.cc:1620] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:38.316442 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=16cae2c9c02347bea55cde7ac221517f] alter complete (version 2)
I20240503 14:13:38.371838 31967 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.376073 31967 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.418751 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:38.425684 31996 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:38.426605 32040 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:38.426884 31994 tablet.cc:1722] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:38.435163 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=16cae2c9c02347bea55cde7ac221517f] alter complete (version 3)
I20240503 14:13:38.538864 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:38.539157 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:38.543743 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet a93d068daa0e466581ce2e44a86a055a
I20240503 14:13:38.544559 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet a93d068daa0e466581ce2e44a86a055a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:38 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:38.545229 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet a93d068daa0e466581ce2e44a86a055a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:38 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:38.545359 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet a93d068daa0e466581ce2e44a86a055a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:38 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:38.545908 32052 tablet_replica.cc:330] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:38.546135 32052 raft_consensus.cc:2229] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:38.546617 32052 raft_consensus.cc:2258] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:38.547717 32052 ts_tablet_manager.cc:1875] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:38.553859 32053 tablet_replica.cc:330] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:38.554076 32054 tablet_replica.cc:330] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:38.554136 32053 raft_consensus.cc:2229] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:38.554322 32054 raft_consensus.cc:2229] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:38.554575 32054 raft_consensus.cc:2258] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:38.554658 32053 raft_consensus.cc:2258] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:38.555104 32052 ts_tablet_manager.cc:1888] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:38.555263 32052 log.cc:1196] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/a93d068daa0e466581ce2e44a86a055a
I20240503 14:13:38.555644 32053 ts_tablet_manager.cc:1875] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:38.555745 32052 ts_tablet_manager.cc:1909] T a93d068daa0e466581ce2e44a86a055a P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:38.555685 32054 ts_tablet_manager.cc:1875] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:38.556972 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet a93d068daa0e466581ce2e44a86a055a (table rebalancer_dynamic_table_new_name [id=16cae2c9c02347bea55cde7ac221517f]) successfully deleted
I20240503 14:13:38.560660 32053 ts_tablet_manager.cc:1888] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:38.560781 32053 log.cc:1196] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/a93d068daa0e466581ce2e44a86a055a
I20240503 14:13:38.560781 32054 ts_tablet_manager.cc:1888] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:38.560894 32054 log.cc:1196] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/a93d068daa0e466581ce2e44a86a055a
I20240503 14:13:38.561203 32053 ts_tablet_manager.cc:1909] T a93d068daa0e466581ce2e44a86a055a P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:38.561239 32054 ts_tablet_manager.cc:1909] T a93d068daa0e466581ce2e44a86a055a P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:38.562067 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet a93d068daa0e466581ce2e44a86a055a (table rebalancer_dynamic_table_new_name [id=16cae2c9c02347bea55cde7ac221517f]) successfully deleted
I20240503 14:13:38.562088 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet a93d068daa0e466581ce2e44a86a055a (table rebalancer_dynamic_table_new_name [id=16cae2c9c02347bea55cde7ac221517f]) successfully deleted
I20240503 14:13:38.593806 32051 consensus_queue.cc:1035] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.598357 32005 consensus_queue.cc:1035] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.647317 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:38.654445 31050 tablet_service.cc:1467] Processing CreateTablet for tablet c521cff52c1f4531a4dc24a625504328 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f44bf89530445e9b3eba0c6bf95d0b9]), partition=
I20240503 14:13:38.654817 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c521cff52c1f4531a4dc24a625504328. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.654966 28414 tablet_service.cc:1467] Processing CreateTablet for tablet c521cff52c1f4531a4dc24a625504328 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f44bf89530445e9b3eba0c6bf95d0b9]), partition=
I20240503 14:13:38.655332 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c521cff52c1f4531a4dc24a625504328. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.658619 31181 tablet_service.cc:1467] Processing CreateTablet for tablet c521cff52c1f4531a4dc24a625504328 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9f44bf89530445e9b3eba0c6bf95d0b9]), partition=
I20240503 14:13:38.658980 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c521cff52c1f4531a4dc24a625504328. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.660704 32058 tablet_bootstrap.cc:492] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:38.663201 32058 tablet_bootstrap.cc:654] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.665688 32060 tablet_bootstrap.cc:492] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:38.667096 32058 tablet_bootstrap.cc:492] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:38.667237 32058 ts_tablet_manager.cc:1367] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 14:13:38.668082 32060 tablet_bootstrap.cc:654] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.668408 32058 raft_consensus.cc:348] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.668763 32058 raft_consensus.cc:374] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.668934 32058 raft_consensus.cc:729] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.669232 32058 consensus_queue.cc:260] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.669606 32061 tablet_bootstrap.cc:492] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:38.671154 32061 tablet_bootstrap.cc:654] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.671272 32058 ts_tablet_manager.cc:1398] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:38.672374 32060 tablet_bootstrap.cc:492] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:38.672526 32060 ts_tablet_manager.cc:1367] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:38.673432 32061 tablet_bootstrap.cc:492] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:38.673629 32061 ts_tablet_manager.cc:1367] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:13:38.673604 32060 raft_consensus.cc:348] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.673825 32060 raft_consensus.cc:374] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.673945 32060 raft_consensus.cc:729] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.674181 32060 consensus_queue.cc:260] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.674844 32060 ts_tablet_manager.cc:1398] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:38.675629 32061 raft_consensus.cc:348] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.675832 32061 raft_consensus.cc:374] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.675936 32061 raft_consensus.cc:729] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.676162 32061 consensus_queue.cc:260] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.676613 32061 ts_tablet_manager.cc:1398] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:38.740113 32047 raft_consensus.cc:482] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.740288 32047 raft_consensus.cc:504] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.740936 32047 leader_election.cc:288] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:38.741362 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c521cff52c1f4531a4dc24a625504328" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:38.741488 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c521cff52c1f4531a4dc24a625504328" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:38.741612 31202 raft_consensus.cc:2454] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:38.741731 28432 raft_consensus.cc:2454] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:38.742146 31008 leader_election.cc:302] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:38.742437 32047 raft_consensus.cc:2781] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.742558 32047 raft_consensus.cc:482] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.742650 32047 raft_consensus.cc:3037] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.744771 32047 raft_consensus.cc:504] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.745227 32047 leader_election.cc:288] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:38.745602 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c521cff52c1f4531a4dc24a625504328" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:38.745630 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c521cff52c1f4531a4dc24a625504328" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:38.745764 31202 raft_consensus.cc:3037] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.745836 28432 raft_consensus.cc:3037] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.747738 31202 raft_consensus.cc:2454] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:38.747910 28432 raft_consensus.cc:2454] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:38.748183 31006 leader_election.cc:302] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:38.748486 32047 raft_consensus.cc:2781] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.748682 32047 raft_consensus.cc:686] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:38.749017 32047 consensus_queue.cc:237] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:38.752099 27898 catalog_manager.cc:5484] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:38.817241 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:38.825124 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 215498be790445b3948b1c5e2823cec4 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.825383 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 215498be790445b3948b1c5e2823cec4 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.825594 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215498be790445b3948b1c5e2823cec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.825573 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 215498be790445b3948b1c5e2823cec4 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.825750 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215498be790445b3948b1c5e2823cec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.825944 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 215498be790445b3948b1c5e2823cec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.830510 32065 tablet_bootstrap.cc:492] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:38.831290 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.831725 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.831867 30788 tablet_service.cc:1467] Processing CreateTablet for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.831956 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.832199 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.832337 32065 tablet_bootstrap.cc:654] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.832360 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.837457 28022 tablet_service.cc:1467] Processing CreateTablet for tablet f7242c3728b84084a738fde7ae32ec0a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.837829 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7242c3728b84084a738fde7ae32ec0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.838042 31050 tablet_service.cc:1467] Processing CreateTablet for tablet f7242c3728b84084a738fde7ae32ec0a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.838354 32060 tablet_bootstrap.cc:492] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:38.838516 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7242c3728b84084a738fde7ae32ec0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.840014 32060 tablet_bootstrap.cc:654] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.844246 32060 tablet_bootstrap.cc:492] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:38.844548 32060 ts_tablet_manager.cc:1367] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:38.845631 32060 raft_consensus.cc:348] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.845932 32060 raft_consensus.cc:374] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.846057 32060 raft_consensus.cc:729] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.846447 32060 consensus_queue.cc:260] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.847079 32060 ts_tablet_manager.cc:1398] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:38.848431 31181 tablet_service.cc:1467] Processing CreateTablet for tablet f7242c3728b84084a738fde7ae32ec0a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.848771 32058 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:38.850136 32068 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:38.850502 32058 tablet_bootstrap.cc:654] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.851768 32068 tablet_bootstrap.cc:654] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.853389 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7242c3728b84084a738fde7ae32ec0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.860786 32058 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:38.861042 32058 ts_tablet_manager.cc:1367] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240503 14:13:38.861937 32058 raft_consensus.cc:348] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.862277 32058 raft_consensus.cc:374] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.862416 32058 raft_consensus.cc:729] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.862793 32058 consensus_queue.cc:260] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.864804 32058 ts_tablet_manager.cc:1398] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:38.869886 32072 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:38.871348 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:38.872048 32067 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:38.872414 32060 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:38.874159 32060 tablet_bootstrap.cc:654] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.874513 32065 tablet_bootstrap.cc:492] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:38.877362 32065 ts_tablet_manager.cc:1367] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.003s
I20240503 14:13:38.878319 32065 raft_consensus.cc:348] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.878794 28434 raft_consensus.cc:1261] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.879546 31202 raft_consensus.cc:1261] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.880329 32065 raft_consensus.cc:374] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.880517 32065 raft_consensus.cc:729] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.880386 31841 consensus_queue.cc:1035] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.880844 32065 consensus_queue.cc:260] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.882061 32067 tablet_bootstrap.cc:654] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.882982 32072 tablet_bootstrap.cc:654] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.885036 32065 ts_tablet_manager.cc:1398] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:38.886809 31841 consensus_queue.cc:1035] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.902065 32072 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:38.902323 32072 ts_tablet_manager.cc:1367] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.000s
I20240503 14:13:38.903141 32072 raft_consensus.cc:348] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:38.903425 32072 raft_consensus.cc:374] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.903553 32072 raft_consensus.cc:729] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.904054 32072 consensus_queue.cc:260] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:38.904829 32072 ts_tablet_manager.cc:1398] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:38.910960 32068 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:38.911141 32068 ts_tablet_manager.cc:1367] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.011s
I20240503 14:13:38.911819 32040 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:38.911427 27900 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet c521cff52c1f4531a4dc24a625504328 (table rebalancer_dynamic_table [id=9f44bf89530445e9b3eba0c6bf95d0b9]). Expected version 1 got 0
I20240503 14:13:38.912143 32068 raft_consensus.cc:348] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:38.912274 32040 tablet.cc:1620] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:38.912351 32068 raft_consensus.cc:374] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.912442 32068 raft_consensus.cc:729] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.912648 32068 consensus_queue.cc:260] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:38.916832 32068 ts_tablet_manager.cc:1398] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:13:38.917150 32068 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:38.918298 32079 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:38.919613 32079 tablet.cc:1620] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:38.924549 32068 tablet_bootstrap.cc:654] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.928848 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=9f44bf89530445e9b3eba0c6bf95d0b9] alter complete (version 1)
I20240503 14:13:38.945326 32060 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:38.947674 32081 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:38.949189 32081 tablet.cc:1620] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:38.949594 27899 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet c521cff52c1f4531a4dc24a625504328 (table rebalancer_dynamic_table [id=9f44bf89530445e9b3eba0c6bf95d0b9]). Expected version 1 got 0
I20240503 14:13:38.949774 32060 ts_tablet_manager.cc:1367] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.077s user 0.014s sys 0.018s
I20240503 14:13:38.951061 32060 raft_consensus.cc:348] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.951309 32060 raft_consensus.cc:374] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.951433 32060 raft_consensus.cc:729] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.951678 32060 consensus_queue.cc:260] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.952271 32060 ts_tablet_manager.cc:1398] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:38.959285 32067 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:38.959483 32067 ts_tablet_manager.cc:1367] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.088s user 0.002s sys 0.027s
I20240503 14:13:38.960278 32067 raft_consensus.cc:348] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:38.960500 32067 raft_consensus.cc:374] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.960604 32067 raft_consensus.cc:729] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.960634 32068 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:38.960814 32068 ts_tablet_manager.cc:1367] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.005s
I20240503 14:13:38.961061 32067 consensus_queue.cc:260] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:38.961640 32067 ts_tablet_manager.cc:1398] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:38.961548 32068 raft_consensus.cc:348] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.961781 32068 raft_consensus.cc:374] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.961858 32068 raft_consensus.cc:729] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.961995 32067 tablet_bootstrap.cc:492] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:38.962090 32068 consensus_queue.cc:260] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.962970 32068 ts_tablet_manager.cc:1398] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:38.964896 32067 tablet_bootstrap.cc:654] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.971285 32067 tablet_bootstrap.cc:492] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:38.971493 32067 ts_tablet_manager.cc:1367] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240503 14:13:38.972273 32067 raft_consensus.cc:348] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.972430 32067 raft_consensus.cc:374] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.972508 32067 raft_consensus.cc:729] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.972725 32067 consensus_queue.cc:260] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.973246 32067 ts_tablet_manager.cc:1398] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:38.993332 32082 raft_consensus.cc:482] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.993484 32082 raft_consensus.cc:504] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.994141 32082 leader_election.cc:288] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:38.994576 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215498be790445b3948b1c5e2823cec4" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:38.994809 31202 raft_consensus.cc:2454] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:38.995227 30744 leader_election.cc:302] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:38.995513 32082 raft_consensus.cc:2781] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.995630 32082 raft_consensus.cc:482] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.995718 32082 raft_consensus.cc:3037] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.997334 31852 raft_consensus.cc:482] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.997522 31852 raft_consensus.cc:504] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:38.998164 31852 leader_election.cc:288] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:38.998539 32082 raft_consensus.cc:504] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:38.998638 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:38.998886 31071 raft_consensus.cc:2454] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:38.999207 32082 leader_election.cc:288] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:38.999145 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:38.999343 31202 raft_consensus.cc:2454] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:38.999365 27978 leader_election.cc:302] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:38.999653 31852 raft_consensus.cc:2781] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.999748 31852 raft_consensus.cc:482] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.999847 31852 raft_consensus.cc:3037] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.000767 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215498be790445b3948b1c5e2823cec4" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:39.000926 31202 raft_consensus.cc:3037] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.002532 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215498be790445b3948b1c5e2823cec4" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:39.002764 28173 raft_consensus.cc:2454] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:39.002828 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "215498be790445b3948b1c5e2823cec4" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:39.003010 28172 raft_consensus.cc:3037] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.003083 31202 raft_consensus.cc:2454] T 215498be790445b3948b1c5e2823cec4 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:39.003604 30744 leader_election.cc:302] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:39.003907 32082 raft_consensus.cc:2781] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.004029 32082 raft_consensus.cc:686] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:39.004292 32082 consensus_queue.cc:237] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.005759 28172 raft_consensus.cc:2454] T 215498be790445b3948b1c5e2823cec4 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:39.005808 31852 raft_consensus.cc:504] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:39.009799 31852 leader_election.cc:288] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:39.009917 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:39.009932 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:39.010108 31071 raft_consensus.cc:3037] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.010165 31202 raft_consensus.cc:3037] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.010712 27899 catalog_manager.cc:5484] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:39.012212 31071 raft_consensus.cc:2454] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:39.012445 31202 raft_consensus.cc:2454] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:39.012812 27978 leader_election.cc:302] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:39.013075 31852 raft_consensus.cc:2781] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.013283 31852 raft_consensus.cc:686] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:39.013561 31852 consensus_queue.cc:237] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:39.016973 27899 catalog_manager.cc:5484] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:39.039372 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:39.047662 32040 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:39.048219 32040 tablet.cc:1620] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:39.050769 32081 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:39.050769 32079 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:39.051231 32081 tablet.cc:1620] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:39.051331 32079 tablet.cc:1620] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:39.061766 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=9f44bf89530445e9b3eba0c6bf95d0b9] alter complete (version 2)
I20240503 14:13:39.091710 31852 raft_consensus.cc:482] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.091893 31852 raft_consensus.cc:504] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:39.092928 31852 leader_election.cc:288] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:39.093056 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344eb942dc3d4188bc9baf3d1ecc4f7a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:39.093299 30809 raft_consensus.cc:2454] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:39.093242 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344eb942dc3d4188bc9baf3d1ecc4f7a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:39.093474 28304 raft_consensus.cc:2454] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:39.093734 27976 leader_election.cc:302] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:39.094020 31852 raft_consensus.cc:2781] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.094128 31852 raft_consensus.cc:482] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.094220 31852 raft_consensus.cc:3037] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.096629 31852 raft_consensus.cc:504] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:39.097272 31852 leader_election.cc:288] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:39.097561 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344eb942dc3d4188bc9baf3d1ecc4f7a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:39.097615 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344eb942dc3d4188bc9baf3d1ecc4f7a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:39.097769 30809 raft_consensus.cc:3037] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.097822 28304 raft_consensus.cc:3037] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.099977 30809 raft_consensus.cc:2454] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:39.100353 27976 leader_election.cc:302] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:39.100472 28304 raft_consensus.cc:2454] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:39.100780 32086 raft_consensus.cc:2781] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.101305 32086 raft_consensus.cc:686] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:39.101586 32086 consensus_queue.cc:237] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:39.104957 27898 catalog_manager.cc:5484] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:39.167366 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:39.175328 32040 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:39.176301 32081 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:39.176472 32079 tablet.cc:1722] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:39.188546 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=9f44bf89530445e9b3eba0c6bf95d0b9] alter complete (version 3)
I20240503 14:13:39.291350 27900 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:39.291600 27900 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:39.294589 27900 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet c521cff52c1f4531a4dc24a625504328
I20240503 14:13:39.295441 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet c521cff52c1f4531a4dc24a625504328 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:39 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:39.295853 31181 tablet_service.cc:1514] Processing DeleteTablet for tablet c521cff52c1f4531a4dc24a625504328 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:39 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:39.296638 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet c521cff52c1f4531a4dc24a625504328 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:39 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:39.296684 32095 tablet_replica.cc:330] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:39.296757 32094 tablet_replica.cc:330] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:39.296928 32095 raft_consensus.cc:2229] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:39.296976 32094 raft_consensus.cc:2229] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:39.297160 32095 raft_consensus.cc:2258] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:39.297487 32094 raft_consensus.cc:2258] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:39.297995 32095 ts_tablet_manager.cc:1875] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:39.298761 32094 ts_tablet_manager.cc:1875] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:39.302379 32095 ts_tablet_manager.cc:1888] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:39.302558 32095 log.cc:1196] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/c521cff52c1f4531a4dc24a625504328
I20240503 14:13:39.302659 32096 tablet_replica.cc:330] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:39.302902 32096 raft_consensus.cc:2229] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:39.303016 32095 ts_tablet_manager.cc:1909] T c521cff52c1f4531a4dc24a625504328 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:39.303136 32096 raft_consensus.cc:2258] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:39.303835 32096 ts_tablet_manager.cc:1875] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:39.304138 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet c521cff52c1f4531a4dc24a625504328 (table rebalancer_dynamic_table_new_name [id=9f44bf89530445e9b3eba0c6bf95d0b9]) successfully deleted
I20240503 14:13:39.305505 32094 ts_tablet_manager.cc:1888] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:39.305635 32094 log.cc:1196] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/c521cff52c1f4531a4dc24a625504328
I20240503 14:13:39.306047 32094 ts_tablet_manager.cc:1909] T c521cff52c1f4531a4dc24a625504328 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:39.306905 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet c521cff52c1f4531a4dc24a625504328 (table rebalancer_dynamic_table_new_name [id=9f44bf89530445e9b3eba0c6bf95d0b9]) successfully deleted
I20240503 14:13:39.308200 32096 ts_tablet_manager.cc:1888] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:39.308307 32096 log.cc:1196] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/c521cff52c1f4531a4dc24a625504328
I20240503 14:13:39.308652 32096 ts_tablet_manager.cc:1909] T c521cff52c1f4531a4dc24a625504328 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:39.309455 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet c521cff52c1f4531a4dc24a625504328 (table rebalancer_dynamic_table_new_name [id=9f44bf89530445e9b3eba0c6bf95d0b9]) successfully deleted
I20240503 14:13:39.397514 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:39.404774 30788 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6a3d7592764c4393b3380b01902408 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907]), partition=
I20240503 14:13:39.404953 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6a3d7592764c4393b3380b01902408 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907]), partition=
I20240503 14:13:39.405184 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6a3d7592764c4393b3380b01902408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.405304 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6a3d7592764c4393b3380b01902408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.405694 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6a3d7592764c4393b3380b01902408 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907]), partition=
I20240503 14:13:39.406616 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6a3d7592764c4393b3380b01902408. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.411278 32067 tablet_bootstrap.cc:492] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:39.411414 32097 tablet_bootstrap.cc:492] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:39.412846 32067 tablet_bootstrap.cc:654] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.412856 32097 tablet_bootstrap.cc:654] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.417599 32067 tablet_bootstrap.cc:492] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:39.417621 32099 tablet_bootstrap.cc:492] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:39.417786 32067 ts_tablet_manager.cc:1367] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20240503 14:13:39.418782 32067 raft_consensus.cc:348] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.419013 32067 raft_consensus.cc:374] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.419103 32067 raft_consensus.cc:729] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.419348 32067 consensus_queue.cc:260] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.419917 32067 ts_tablet_manager.cc:1398] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:39.420096 32099 tablet_bootstrap.cc:654] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.421346 32097 tablet_bootstrap.cc:492] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:39.421541 32097 ts_tablet_manager.cc:1367] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240503 14:13:39.422401 32097 raft_consensus.cc:348] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.422698 32097 raft_consensus.cc:374] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.422853 32097 raft_consensus.cc:729] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.423128 32097 consensus_queue.cc:260] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.424211 32097 ts_tablet_manager.cc:1398] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:39.424286 32099 tablet_bootstrap.cc:492] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:39.424443 32099 ts_tablet_manager.cc:1367] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240503 14:13:39.424961 32099 raft_consensus.cc:348] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.425122 32099 raft_consensus.cc:374] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.425259 32099 raft_consensus.cc:729] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.425470 32099 consensus_queue.cc:260] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.425910 32099 ts_tablet_manager.cc:1398] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:13:39.528923 32089 consensus_queue.cc:1035] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.529357 32082 consensus_queue.cc:1035] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.535328 32089 consensus_queue.cc:1035] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.548030 32082 consensus_queue.cc:1035] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.575659 32086 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.580907 32089 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.636593 32037 raft_consensus.cc:482] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.636796 32037 raft_consensus.cc:504] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.637470 32037 leader_election.cc:288] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:39.637989 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6a3d7592764c4393b3380b01902408" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:39.638144 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6a3d7592764c4393b3380b01902408" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:39.638417 30940 raft_consensus.cc:2454] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:39.638211 30809 raft_consensus.cc:2454] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:39.638881 28367 leader_election.cc:302] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:39.639168 32037 raft_consensus.cc:2781] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.639278 32037 raft_consensus.cc:482] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.639372 32037 raft_consensus.cc:3037] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.641523 32037 raft_consensus.cc:504] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.642469 32037 leader_election.cc:288] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:39.642627 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6a3d7592764c4393b3380b01902408" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:39.642824 30940 raft_consensus.cc:3037] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.643489 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:39.645543 30940 raft_consensus.cc:2454] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:39.646414 28369 leader_election.cc:302] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:39.647218 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6a3d7592764c4393b3380b01902408" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:39.647631 30809 raft_consensus.cc:3037] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.648900 32038 raft_consensus.cc:2781] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.653910 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 645bb849069a4f65930d825edd532cb0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.654388 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645bb849069a4f65930d825edd532cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.654345 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 645bb849069a4f65930d825edd532cb0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.654688 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645bb849069a4f65930d825edd532cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.659232 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 645bb849069a4f65930d825edd532cb0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.659417 31181 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb66488d714d39900331f4a257e8b3 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.659658 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645bb849069a4f65930d825edd532cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.659858 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb66488d714d39900331f4a257e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.661011 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 5be4220923ed4c5388a7a15755cb685d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.661401 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be4220923ed4c5388a7a15755cb685d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.662904 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 5be4220923ed4c5388a7a15755cb685d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.662905 30919 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb66488d714d39900331f4a257e8b3 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.663302 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb66488d714d39900331f4a257e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.663321 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be4220923ed4c5388a7a15755cb685d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.664567 30809 raft_consensus.cc:2454] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:39.665480 30788 tablet_service.cc:1467] Processing CreateTablet for tablet d2bb66488d714d39900331f4a257e8b3 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.665875 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb66488d714d39900331f4a257e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.667148 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 5be4220923ed4c5388a7a15755cb685d (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.667490 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be4220923ed4c5388a7a15755cb685d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.673952 32117 tablet_bootstrap.cc:492] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:39.675613 32097 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:39.675915 32117 tablet_bootstrap.cc:654] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.677634 32097 tablet_bootstrap.cc:654] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.688263 32067 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:39.691278 32067 tablet_bootstrap.cc:654] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.698355 32038 raft_consensus.cc:686] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:39.698729 32038 consensus_queue.cc:237] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:39.704442 32099 tablet_bootstrap.cc:492] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:39.706569 32099 tablet_bootstrap.cc:654] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.710347 32117 tablet_bootstrap.cc:492] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:39.711760 32123 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:39.712370 32117 ts_tablet_manager.cc:1367] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.011s
I20240503 14:13:39.713820 32117 raft_consensus.cc:348] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.716620 32117 raft_consensus.cc:374] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.717042 32117 raft_consensus.cc:729] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.718228 32117 consensus_queue.cc:260] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.718647 32099 tablet_bootstrap.cc:492] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:39.718881 32099 ts_tablet_manager.cc:1367] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240503 14:13:39.719540 32117 ts_tablet_manager.cc:1398] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:39.719681 32099 raft_consensus.cc:348] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.720009 32099 raft_consensus.cc:374] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.720132 32099 raft_consensus.cc:729] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.720449 32099 consensus_queue.cc:260] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.721159 32099 ts_tablet_manager.cc:1398] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:39.721865 32114 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
W20240503 14:13:39.723205 28109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 52238.4us
I20240503 14:13:39.724330 32114 tablet_bootstrap.cc:654] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.731346 32123 tablet_bootstrap.cc:654] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.739678 32114 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:39.739746 32067 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:39.739935 32114 ts_tablet_manager.cc:1367] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20240503 14:13:39.739966 32067 ts_tablet_manager.cc:1367] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.011s
I20240503 14:13:39.740065 27900 catalog_manager.cc:5484] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:39.740794 32097 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:39.741030 32097 ts_tablet_manager.cc:1367] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.015s
I20240503 14:13:39.740972 32114 raft_consensus.cc:348] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.741247 32114 raft_consensus.cc:374] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.741406 32114 raft_consensus.cc:729] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.741714 32114 consensus_queue.cc:260] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.741999 32097 raft_consensus.cc:348] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.742354 32097 raft_consensus.cc:374] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.742491 32097 raft_consensus.cc:729] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.742851 32097 consensus_queue.cc:260] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.743541 32114 ts_tablet_manager.cc:1398] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:39.743849 32114 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:39.743885 32097 ts_tablet_manager.cc:1398] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:39.746829 32114 tablet_bootstrap.cc:654] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.747115 32123 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:39.747277 32123 ts_tablet_manager.cc:1367] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.007s
I20240503 14:13:39.747568 32067 raft_consensus.cc:348] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.747924 32067 raft_consensus.cc:374] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.748021 32067 raft_consensus.cc:729] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.748284 32067 consensus_queue.cc:260] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.749418 32123 raft_consensus.cc:348] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.749696 32123 raft_consensus.cc:374] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.749800 32123 raft_consensus.cc:729] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.750407 32123 consensus_queue.cc:260] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.751145 32123 ts_tablet_manager.cc:1398] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:13:39.751504 32123 tablet_bootstrap.cc:492] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:39.753654 32123 tablet_bootstrap.cc:654] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.756078 32123 tablet_bootstrap.cc:492] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:39.756242 32123 ts_tablet_manager.cc:1367] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 14:13:39.756650 32067 ts_tablet_manager.cc:1398] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20240503 14:13:39.756973 32067 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:39.756980 32123 raft_consensus.cc:348] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.757215 32123 raft_consensus.cc:374] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.757304 32123 raft_consensus.cc:729] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.757547 32123 consensus_queue.cc:260] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.758283 32123 ts_tablet_manager.cc:1398] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:13:39.759068 32067 tablet_bootstrap.cc:654] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.763218 32114 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:39.763504 32114 ts_tablet_manager.cc:1367] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.011s
I20240503 14:13:39.764448 32114 raft_consensus.cc:348] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.764693 32114 raft_consensus.cc:374] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.764776 32114 raft_consensus.cc:729] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.765069 32114 consensus_queue.cc:260] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.765272 32067 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:39.765439 32067 ts_tablet_manager.cc:1367] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:13:39.766155 32067 raft_consensus.cc:348] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.766371 32067 raft_consensus.cc:374] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.766456 32067 raft_consensus.cc:729] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.766548 32114 ts_tablet_manager.cc:1398] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:39.766661 32067 consensus_queue.cc:260] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.767324 32067 ts_tablet_manager.cc:1398] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:39.824882 31940 raft_consensus.cc:482] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.825086 31940 raft_consensus.cc:504] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.825801 31940 leader_election.cc:288] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:39.826213 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb66488d714d39900331f4a257e8b3" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:39.826279 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb66488d714d39900331f4a257e8b3" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:39.826488 30940 raft_consensus.cc:2454] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:39.826512 30809 raft_consensus.cc:2454] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:39.826949 31136 leader_election.cc:302] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:39.827231 31940 raft_consensus.cc:2781] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.827345 31940 raft_consensus.cc:482] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.827443 31940 raft_consensus.cc:3037] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.829485 31940 raft_consensus.cc:504] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.830001 31940 leader_election.cc:288] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:39.830507 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb66488d714d39900331f4a257e8b3" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:39.830592 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2bb66488d714d39900331f4a257e8b3" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:39.830729 30809 raft_consensus.cc:3037] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.830782 30940 raft_consensus.cc:3037] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.833493 30809 raft_consensus.cc:2454] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:39.833510 30940 raft_consensus.cc:2454] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:39.833897 31136 leader_election.cc:302] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:39.834282 31940 raft_consensus.cc:2781] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.834674 31940 raft_consensus.cc:686] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:39.834993 31940 consensus_queue.cc:237] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:39.838354 27901 catalog_manager.cc:5484] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:39.878707 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:39.885766 30809 raft_consensus.cc:1261] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:39.885820 30940 raft_consensus.cc:1261] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:39.886760 32037 consensus_queue.cc:1035] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.887285 32037 consensus_queue.cc:1035] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.900676 27901 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 8a6a3d7592764c4393b3380b01902408 (table rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907]). Expected version 1 got 0
I20240503 14:13:39.901506 27899 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 8a6a3d7592764c4393b3380b01902408 (table rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907]). Expected version 1 got 0
I20240503 14:13:39.902333 32057 raft_consensus.cc:482] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.902498 32057 raft_consensus.cc:504] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.903524 32057 leader_election.cc:288] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:39.903836 32137 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:39.904173 32037 raft_consensus.cc:482] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.904198 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645bb849069a4f65930d825edd532cb0" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:39.904323 32037 raft_consensus.cc:504] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.904629 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645bb849069a4f65930d825edd532cb0" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:39.904742 32137 tablet.cc:1620] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:39.904855 31071 raft_consensus.cc:2454] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:39.905145 28434 raft_consensus.cc:2454] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:39.905997 28109 leader_election.cc:302] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:39.906344 32057 raft_consensus.cc:2781] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.906467 32057 raft_consensus.cc:482] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.906602 32057 raft_consensus.cc:3037] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.911674 32138 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:39.912371 32138 tablet.cc:1620] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:39.912371 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645bb849069a4f65930d825edd532cb0" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:39.912686 32037 leader_election.cc:288] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:39.913067 27899 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 8a6a3d7592764c4393b3380b01902408 (table rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907]). Expected version 1 got 0
I20240503 14:13:39.913340 32057 raft_consensus.cc:504] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.913750 28173 raft_consensus.cc:2379] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5388902973134958a030fd63ab4faba4 in current term 1: Already voted for candidate 6409961ee8c0445687c11724213c44a9 in this term.
I20240503 14:13:39.920477 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907] alter complete (version 1)
I20240503 14:13:39.921645 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645bb849069a4f65930d825edd532cb0" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:39.921916 31071 raft_consensus.cc:3037] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.925199 31071 raft_consensus.cc:2454] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:39.925806 28109 leader_election.cc:302] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:39.926213 32115 raft_consensus.cc:2781] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.926422 32115 raft_consensus.cc:686] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:39.926829 32115 consensus_queue.cc:237] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:39.933646 32057 leader_election.cc:288] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:39.942816 32141 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:39.943456 32141 tablet.cc:1620] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:39.944887 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645bb849069a4f65930d825edd532cb0" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:39.945132 28434 raft_consensus.cc:3037] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.946856 27899 catalog_manager.cc:5484] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:39.948246 28434 raft_consensus.cc:2454] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:39.950278 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645bb849069a4f65930d825edd532cb0" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:39.950544 31071 raft_consensus.cc:2379] T 645bb849069a4f65930d825edd532cb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5388902973134958a030fd63ab4faba4 in current term 1: Already voted for candidate 6409961ee8c0445687c11724213c44a9 in this term.
I20240503 14:13:39.950946 28369 leader_election.cc:302] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:39.951509 32037 raft_consensus.cc:2726] T 645bb849069a4f65930d825edd532cb0 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:39.952184 27899 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 8a6a3d7592764c4393b3380b01902408 (table rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907]). Expected version 1 got 0
I20240503 14:13:40.021981 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:40.029165 32137 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.029685 32137 tablet.cc:1620] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:40.030441 32138 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.030915 32138 tablet.cc:1620] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:40.038199 32141 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.038690 32141 tablet.cc:1620] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:40.039834 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=88b12ef5505e4eb0a819792006134907] alter complete (version 2)
I20240503 14:13:40.065418 32082 raft_consensus.cc:482] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.065641 32082 raft_consensus.cc:504] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.066138 32082 leader_election.cc:288] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:40.066663 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4220923ed4c5388a7a15755cb685d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:40.066838 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4220923ed4c5388a7a15755cb685d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:40.066921 31071 raft_consensus.cc:2454] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:40.067109 31203 raft_consensus.cc:2454] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:40.067415 30745 leader_election.cc:302] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:40.067705 32082 raft_consensus.cc:2781] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.067782 32082 raft_consensus.cc:482] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.067854 32082 raft_consensus.cc:3037] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.069722 32082 raft_consensus.cc:504] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.070300 32082 leader_election.cc:288] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:40.070668 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4220923ed4c5388a7a15755cb685d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:40.070864 31203 raft_consensus.cc:3037] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.070822 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be4220923ed4c5388a7a15755cb685d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:40.071028 31071 raft_consensus.cc:3037] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.072691 31203 raft_consensus.cc:2454] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:40.072885 31071 raft_consensus.cc:2454] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:40.073314 30745 leader_election.cc:302] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:40.073571 32082 raft_consensus.cc:2781] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.074191 32082 raft_consensus.cc:686] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:40.074541 32082 consensus_queue.cc:237] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.077816 27899 catalog_manager.cc:5484] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:40.143324 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:40.151491 32137 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:40.151671 32138 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:40.152079 32141 tablet.cc:1722] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:40.161777 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=88b12ef5505e4eb0a819792006134907] alter complete (version 3)
I20240503 14:13:40.264489 27901 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:40.264704 27901 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:40.267508 27901 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 8a6a3d7592764c4393b3380b01902408
I20240503 14:13:40.268311 28414 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6a3d7592764c4393b3380b01902408 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:40 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:40.268607 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6a3d7592764c4393b3380b01902408 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:40 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:40.268893 32147 tablet_replica.cc:330] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:40.268934 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6a3d7592764c4393b3380b01902408 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:40 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:40.269196 32147 raft_consensus.cc:2229] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:40.269263 32148 tablet_replica.cc:330] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:40.269447 32149 tablet_replica.cc:330] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:40.269510 32148 raft_consensus.cc:2229] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.269538 32147 raft_consensus.cc:2258] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.269652 32149 raft_consensus.cc:2229] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.269752 32148 raft_consensus.cc:2258] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.269865 32149 raft_consensus.cc:2258] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.270442 32147 ts_tablet_manager.cc:1875] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:40.270622 32149 ts_tablet_manager.cc:1875] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:40.270630 32148 ts_tablet_manager.cc:1875] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:40.276094 32149 ts_tablet_manager.cc:1888] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:40.276238 32149 log.cc:1196] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/8a6a3d7592764c4393b3380b01902408
I20240503 14:13:40.276783 32149 ts_tablet_manager.cc:1909] T 8a6a3d7592764c4393b3380b01902408 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:40.276888 32148 ts_tablet_manager.cc:1888] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:40.277026 32148 log.cc:1196] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/8a6a3d7592764c4393b3380b01902408
I20240503 14:13:40.277441 32148 ts_tablet_manager.cc:1909] T 8a6a3d7592764c4393b3380b01902408 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:40.277455 32147 ts_tablet_manager.cc:1888] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:40.277545 32147 log.cc:1196] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a6a3d7592764c4393b3380b01902408
I20240503 14:13:40.277869 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 8a6a3d7592764c4393b3380b01902408 (table rebalancer_dynamic_table_new_name [id=88b12ef5505e4eb0a819792006134907]) successfully deleted
I20240503 14:13:40.277948 32147 ts_tablet_manager.cc:1909] T 8a6a3d7592764c4393b3380b01902408 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:40.278447 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 8a6a3d7592764c4393b3380b01902408 (table rebalancer_dynamic_table_new_name [id=88b12ef5505e4eb0a819792006134907]) successfully deleted
I20240503 14:13:40.278869 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 8a6a3d7592764c4393b3380b01902408 (table rebalancer_dynamic_table_new_name [id=88b12ef5505e4eb0a819792006134907]) successfully deleted
I20240503 14:13:40.280261 31940 consensus_queue.cc:1035] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.285182 31940 consensus_queue.cc:1035] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.369851 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:40.374714 32142 consensus_queue.cc:1035] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.379792 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 3d5c8eb14f024be9bcd05ad4993c8129 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2]), partition=
I20240503 14:13:40.380074 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 3d5c8eb14f024be9bcd05ad4993c8129 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2]), partition=
I20240503 14:13:40.380199 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d5c8eb14f024be9bcd05ad4993c8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.380421 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d5c8eb14f024be9bcd05ad4993c8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.385000 32157 tablet_bootstrap.cc:492] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:40.385344 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 3d5c8eb14f024be9bcd05ad4993c8129 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2]), partition=
I20240503 14:13:40.385725 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d5c8eb14f024be9bcd05ad4993c8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.386777 32157 tablet_bootstrap.cc:654] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.387257 32142 consensus_queue.cc:1035] T 645bb849069a4f65930d825edd532cb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.403890 32159 tablet_bootstrap.cc:492] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:40.411192 32159 tablet_bootstrap.cc:654] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.412652 32162 tablet_bootstrap.cc:492] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:40.415326 32162 tablet_bootstrap.cc:654] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.423897 32157 tablet_bootstrap.cc:492] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:40.424101 32157 ts_tablet_manager.cc:1367] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.021s
I20240503 14:13:40.425078 32157 raft_consensus.cc:348] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.425382 32157 raft_consensus.cc:374] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.425501 32157 raft_consensus.cc:729] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.425833 32157 consensus_queue.cc:260] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.428040 32157 ts_tablet_manager.cc:1398] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:13:40.428740 32159 tablet_bootstrap.cc:492] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:40.428781 32162 tablet_bootstrap.cc:492] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:40.428961 32162 ts_tablet_manager.cc:1367] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240503 14:13:40.428970 32159 ts_tablet_manager.cc:1367] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.008s
I20240503 14:13:40.429761 32159 raft_consensus.cc:348] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.429790 32162 raft_consensus.cc:348] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.430018 32162 raft_consensus.cc:374] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.430052 32159 raft_consensus.cc:374] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.430117 32162 raft_consensus.cc:729] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.430195 32159 raft_consensus.cc:729] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.430531 32159 consensus_queue.cc:260] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.430383 32162 consensus_queue.cc:260] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.431113 32162 ts_tablet_manager.cc:1398] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:40.431756 32159 ts_tablet_manager.cc:1398] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:40.505662 32082 consensus_queue.cc:1035] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.510852 32145 consensus_queue.cc:1035] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.562127 31967 raft_consensus.cc:482] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.562356 31967 raft_consensus.cc:504] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.563017 31967 leader_election.cc:288] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:40.563457 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d5c8eb14f024be9bcd05ad4993c8129" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:40.563571 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d5c8eb14f024be9bcd05ad4993c8129" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:40.563712 30939 raft_consensus.cc:2454] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:40.563808 30809 raft_consensus.cc:2454] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:40.564152 31007 leader_election.cc:302] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:40.564461 31967 raft_consensus.cc:2781] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.564538 31967 raft_consensus.cc:482] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.564635 31967 raft_consensus.cc:3037] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.566799 31967 raft_consensus.cc:504] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.567371 31967 leader_election.cc:288] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:40.567730 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d5c8eb14f024be9bcd05ad4993c8129" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:40.567842 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d5c8eb14f024be9bcd05ad4993c8129" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:40.567951 30939 raft_consensus.cc:3037] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.568073 30809 raft_consensus.cc:3037] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.569993 30939 raft_consensus.cc:2454] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:40.570384 31007 leader_election.cc:302] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:40.570560 30809 raft_consensus.cc:2454] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:40.570731 31967 raft_consensus.cc:2781] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.570976 31967 raft_consensus.cc:686] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:40.571231 31967 consensus_queue.cc:237] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.575964 27901 catalog_manager.cc:5484] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:40.652866 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:40.661324 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 619951b364d44766a2f3064222ceb738 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.661582 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 619951b364d44766a2f3064222ceb738 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.661799 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619951b364d44766a2f3064222ceb738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.661962 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619951b364d44766a2f3064222ceb738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.663017 30788 tablet_service.cc:1467] Processing CreateTablet for tablet d9119ae1f5ca4c12ba7a4444edd210cf (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.663357 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9119ae1f5ca4c12ba7a4444edd210cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.666653 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 619951b364d44766a2f3064222ceb738 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.667083 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619951b364d44766a2f3064222ceb738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.668335 28023 tablet_service.cc:1467] Processing CreateTablet for tablet d9119ae1f5ca4c12ba7a4444edd210cf (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.668432 31182 tablet_service.cc:1467] Processing CreateTablet for tablet d9119ae1f5ca4c12ba7a4444edd210cf (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.668789 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9119ae1f5ca4c12ba7a4444edd210cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.668838 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9119ae1f5ca4c12ba7a4444edd210cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.668988 31181 tablet_service.cc:1467] Processing CreateTablet for tablet bf316cde5631415faecbc1bb26690b79 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.669373 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf316cde5631415faecbc1bb26690b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.676975 28414 tablet_service.cc:1467] Processing CreateTablet for tablet bf316cde5631415faecbc1bb26690b79 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.677532 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf316cde5631415faecbc1bb26690b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.677560 32169 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:40.678027 32162 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:40.679101 32169 tablet_bootstrap.cc:654] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.679597 32162 tablet_bootstrap.cc:654] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.685261 30920 tablet_service.cc:1467] Processing CreateTablet for tablet bf316cde5631415faecbc1bb26690b79 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.685696 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf316cde5631415faecbc1bb26690b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.690977 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:40.693244 32159 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:40.694921 32159 tablet_bootstrap.cc:654] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.705224 30810 raft_consensus.cc:1261] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.705724 30940 raft_consensus.cc:1261] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.706369 31967 consensus_queue.cc:1035] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.706676 32152 consensus_queue.cc:1035] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.710397 32168 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:40.712375 27900 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 3d5c8eb14f024be9bcd05ad4993c8129 (table rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2]). Expected version 1 got 0
I20240503 14:13:40.712523 32168 tablet_bootstrap.cc:654] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.715023 32159 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:40.715291 32159 ts_tablet_manager.cc:1367] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.002s
I20240503 14:13:40.716132 32159 raft_consensus.cc:348] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.716426 32159 raft_consensus.cc:374] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.716567 32159 raft_consensus.cc:729] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.716882 32159 consensus_queue.cc:260] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.717542 32159 ts_tablet_manager.cc:1398] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:40.717962 32159 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:40.719100 32177 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:40.719957 32177 tablet.cc:1620] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:40.720351 32169 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:40.720600 32169 ts_tablet_manager.cc:1367] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.043s user 0.013s sys 0.018s
I20240503 14:13:40.721701 32169 raft_consensus.cc:348] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.722178 32169 raft_consensus.cc:374] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.722365 32169 raft_consensus.cc:729] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.722712 32169 consensus_queue.cc:260] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.723481 32169 ts_tablet_manager.cc:1398] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:40.724452 32159 tablet_bootstrap.cc:654] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.734453 32162 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:40.734678 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2] alter complete (version 1)
I20240503 14:13:40.735210 32162 ts_tablet_manager.cc:1367] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240503 14:13:40.736068 32162 raft_consensus.cc:348] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.736243 32162 raft_consensus.cc:374] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.738772 32162 raft_consensus.cc:729] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.738577 32178 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:40.740732 32178 tablet.cc:1620] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:40.742081 27899 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 3d5c8eb14f024be9bcd05ad4993c8129 (table rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2]). Expected version 1 got 0
I20240503 14:13:40.743499 27901 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 3d5c8eb14f024be9bcd05ad4993c8129 (table rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2]). Expected version 1 got 0
I20240503 14:13:40.743752 32162 consensus_queue.cc:260] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.744786 32162 ts_tablet_manager.cc:1398] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:13:40.746120 32181 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:40.747046 32181 tablet.cc:1620] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:40.747900 32162 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:40.751565 32162 tablet_bootstrap.cc:654] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.752161 32173 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:40.755064 32173 tablet_bootstrap.cc:654] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.764348 32168 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:40.764637 32168 ts_tablet_manager.cc:1367] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.004s
I20240503 14:13:40.767238 32168 raft_consensus.cc:348] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.767683 32168 raft_consensus.cc:374] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.767844 32168 raft_consensus.cc:729] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.768203 32168 consensus_queue.cc:260] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.769917 32159 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:40.770073 32159 ts_tablet_manager.cc:1367] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.052s user 0.017s sys 0.028s
I20240503 14:13:40.771047 32168 ts_tablet_manager.cc:1398] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:40.770952 32159 raft_consensus.cc:348] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.771155 32173 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:40.771189 32159 raft_consensus.cc:374] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.771328 32159 raft_consensus.cc:729] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.771445 32168 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:40.771483 32173 ts_tablet_manager.cc:1367] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20240503 14:13:40.771601 32159 consensus_queue.cc:260] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.773083 32159 ts_tablet_manager.cc:1398] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:40.773005 32173 raft_consensus.cc:348] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.774161 32168 tablet_bootstrap.cc:654] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.774667 32162 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:40.774880 32162 ts_tablet_manager.cc:1367] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20240503 14:13:40.776018 32162 raft_consensus.cc:348] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.776298 32162 raft_consensus.cc:374] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.776446 32162 raft_consensus.cc:729] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.776741 32162 consensus_queue.cc:260] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.780053 32162 ts_tablet_manager.cc:1398] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:40.773430 32173 raft_consensus.cc:374] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.780318 32173 raft_consensus.cc:729] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.780561 32173 consensus_queue.cc:260] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.781281 32168 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:40.781445 32168 ts_tablet_manager.cc:1367] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240503 14:13:40.782264 32173 ts_tablet_manager.cc:1398] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 14:13:40.783680 32168 raft_consensus.cc:348] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.783923 32168 raft_consensus.cc:374] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.784008 32168 raft_consensus.cc:729] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.784287 32168 consensus_queue.cc:260] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.785059 32168 ts_tablet_manager.cc:1398] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:40.785369 32168 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:40.788585 32168 tablet_bootstrap.cc:654] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.790416 32168 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:40.790539 32168 ts_tablet_manager.cc:1367] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:13:40.791126 32168 raft_consensus.cc:348] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.791285 32168 raft_consensus.cc:374] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.791376 32168 raft_consensus.cc:729] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.791569 32168 consensus_queue.cc:260] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.791996 32168 ts_tablet_manager.cc:1398] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:40.839061 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:40.841998 32145 raft_consensus.cc:482] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.842178 32145 raft_consensus.cc:504] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.842857 32145 leader_election.cc:288] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:40.843958 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9119ae1f5ca4c12ba7a4444edd210cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:40.844157 28042 raft_consensus.cc:2454] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:40.844290 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9119ae1f5ca4c12ba7a4444edd210cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:40.844527 31202 raft_consensus.cc:2454] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:40.844839 30746 leader_election.cc:302] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:40.845158 32145 raft_consensus.cc:2781] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.845269 32145 raft_consensus.cc:482] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.845378 32145 raft_consensus.cc:3037] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.847563 32177 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.848093 32177 tablet.cc:1620] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:40.848526 32178 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.848523 32181 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.848940 32178 tablet.cc:1620] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:40.848950 32181 tablet.cc:1620] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:40.850206 32145 raft_consensus.cc:504] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.851137 32145 leader_election.cc:288] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:40.851167 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9119ae1f5ca4c12ba7a4444edd210cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:40.851378 28042 raft_consensus.cc:3037] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.851511 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9119ae1f5ca4c12ba7a4444edd210cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:40.851688 31202 raft_consensus.cc:3037] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.854125 28042 raft_consensus.cc:2454] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:40.854441 31202 raft_consensus.cc:2454] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:40.857758 30746 leader_election.cc:302] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:40.862932 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=48c9cfc633c645dd8590f038fb340be2] alter complete (version 2)
I20240503 14:13:40.860723 32145 raft_consensus.cc:2781] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.866586 32145 raft_consensus.cc:686] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:40.866886 32145 consensus_queue.cc:237] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:40.871320 27901 catalog_manager.cc:5484] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:40.877825 32154 raft_consensus.cc:482] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.877995 32154 raft_consensus.cc:504] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.878687 32154 leader_election.cc:288] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:40.879097 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619951b364d44766a2f3064222ceb738" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:40.879105 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619951b364d44766a2f3064222ceb738" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:40.879276 30809 raft_consensus.cc:2454] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:40.879329 31202 raft_consensus.cc:2454] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:40.879678 30874 leader_election.cc:302] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:40.881645 32154 raft_consensus.cc:2781] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.881757 32154 raft_consensus.cc:482] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.881832 32154 raft_consensus.cc:3037] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.884434 32154 raft_consensus.cc:504] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.889582 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619951b364d44766a2f3064222ceb738" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:40.889792 31202 raft_consensus.cc:3037] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.890578 32154 leader_election.cc:288] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:40.892607 31202 raft_consensus.cc:2454] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:40.892946 31940 raft_consensus.cc:482] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.893090 31940 raft_consensus.cc:504] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.893745 31940 leader_election.cc:288] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:40.894902 30875 leader_election.cc:302] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:40.895164 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf316cde5631415faecbc1bb26690b79" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:40.895395 30939 raft_consensus.cc:2454] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:40.895686 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619951b364d44766a2f3064222ceb738" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:40.895906 30809 raft_consensus.cc:3037] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.896111 31138 leader_election.cc:302] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:40.896449 31940 raft_consensus.cc:2781] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.896562 31940 raft_consensus.cc:482] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.896653 31940 raft_consensus.cc:3037] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.899353 31940 raft_consensus.cc:504] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.900322 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf316cde5631415faecbc1bb26690b79" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:40.900480 30939 raft_consensus.cc:3037] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.901373 30809 raft_consensus.cc:2454] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:40.903438 32116 raft_consensus.cc:2781] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.904808 30939 raft_consensus.cc:2454] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:40.905205 31138 leader_election.cc:302] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:40.909399 32191 raft_consensus.cc:2781] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.909564 31940 leader_election.cc:288] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:40.912535 32191 raft_consensus.cc:686] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:40.912864 32191 consensus_queue.cc:237] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:40.917973 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf316cde5631415faecbc1bb26690b79" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:40.918036 32116 raft_consensus.cc:686] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:40.918010 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf316cde5631415faecbc1bb26690b79" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:40.918210 28432 raft_consensus.cc:3037] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.918426 32116 consensus_queue.cc:237] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:40.921095 28432 raft_consensus.cc:2454] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:40.922083 27901 catalog_manager.cc:5484] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:40.930564 27901 catalog_manager.cc:5484] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:40.965842 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:40.973472 32177 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:40.974298 32178 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:40.974474 32181 tablet.cc:1722] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:40.983757 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=48c9cfc633c645dd8590f038fb340be2] alter complete (version 3)
I20240503 14:13:41.087031 27901 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:41.087275 27901 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:41.089960 27901 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 3d5c8eb14f024be9bcd05ad4993c8129
I20240503 14:13:41.091017 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d5c8eb14f024be9bcd05ad4993c8129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:41.091197 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d5c8eb14f024be9bcd05ad4993c8129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:41.091609 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d5c8eb14f024be9bcd05ad4993c8129 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:41.091830 32197 tablet_replica.cc:330] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:41.092121 32197 raft_consensus.cc:2229] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.092339 32198 tablet_replica.cc:330] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:41.092346 32197 raft_consensus.cc:2258] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.092535 32198 raft_consensus.cc:2229] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:41.093053 32198 raft_consensus.cc:2258] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.093380 32197 ts_tablet_manager.cc:1875] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.094018 32196 tablet_replica.cc:330] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:41.094213 32196 raft_consensus.cc:2229] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.094345 32198 ts_tablet_manager.cc:1875] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.094452 32196 raft_consensus.cc:2258] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.095165 32196 ts_tablet_manager.cc:1875] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.099108 32196 ts_tablet_manager.cc:1888] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.099217 32196 log.cc:1196] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/3d5c8eb14f024be9bcd05ad4993c8129
I20240503 14:13:41.099666 32196 ts_tablet_manager.cc:1909] T 3d5c8eb14f024be9bcd05ad4993c8129 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:41.099653 32197 ts_tablet_manager.cc:1888] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.099763 32197 log.cc:1196] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/3d5c8eb14f024be9bcd05ad4993c8129
I20240503 14:13:41.100226 32197 ts_tablet_manager.cc:1909] T 3d5c8eb14f024be9bcd05ad4993c8129 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:41.100258 32198 ts_tablet_manager.cc:1888] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.100355 32198 log.cc:1196] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/3d5c8eb14f024be9bcd05ad4993c8129
I20240503 14:13:41.100503 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 3d5c8eb14f024be9bcd05ad4993c8129 (table rebalancer_dynamic_table_new_name [id=48c9cfc633c645dd8590f038fb340be2]) successfully deleted
I20240503 14:13:41.100749 32198 ts_tablet_manager.cc:1909] T 3d5c8eb14f024be9bcd05ad4993c8129 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:41.101279 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 3d5c8eb14f024be9bcd05ad4993c8129 (table rebalancer_dynamic_table_new_name [id=48c9cfc633c645dd8590f038fb340be2]) successfully deleted
I20240503 14:13:41.101708 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 3d5c8eb14f024be9bcd05ad4993c8129 (table rebalancer_dynamic_table_new_name [id=48c9cfc633c645dd8590f038fb340be2]) successfully deleted
I20240503 14:13:41.192612 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:41.200026 30919 tablet_service.cc:1467] Processing CreateTablet for tablet d2e62601bbbf4686a68333c4acd8cd05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3]), partition=
I20240503 14:13:41.200112 31183 tablet_service.cc:1467] Processing CreateTablet for tablet d2e62601bbbf4686a68333c4acd8cd05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3]), partition=
I20240503 14:13:41.200439 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e62601bbbf4686a68333c4acd8cd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.200470 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e62601bbbf4686a68333c4acd8cd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.206857 28282 tablet_service.cc:1467] Processing CreateTablet for tablet d2e62601bbbf4686a68333c4acd8cd05 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3]), partition=
I20240503 14:13:41.207283 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2e62601bbbf4686a68333c4acd8cd05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.207643 32168 tablet_bootstrap.cc:492] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:41.209206 32168 tablet_bootstrap.cc:654] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.210074 32159 tablet_bootstrap.cc:492] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:41.211616 32159 tablet_bootstrap.cc:654] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.220175 32201 tablet_bootstrap.cc:492] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:41.222486 32201 tablet_bootstrap.cc:654] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.222694 32159 tablet_bootstrap.cc:492] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:41.222870 32168 tablet_bootstrap.cc:492] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:41.222880 32159 ts_tablet_manager.cc:1367] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:13:41.223031 32168 ts_tablet_manager.cc:1367] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.005s
I20240503 14:13:41.223789 32159 raft_consensus.cc:348] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.223968 32168 raft_consensus.cc:348] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.224090 32159 raft_consensus.cc:374] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.224171 32159 raft_consensus.cc:729] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.224200 32168 raft_consensus.cc:374] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.224272 32168 raft_consensus.cc:729] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.224534 32168 consensus_queue.cc:260] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.225219 32168 ts_tablet_manager.cc:1398] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:41.224562 32159 consensus_queue.cc:260] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.226825 32159 ts_tablet_manager.cc:1398] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:41.233042 32201 tablet_bootstrap.cc:492] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:41.233269 32201 ts_tablet_manager.cc:1367] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240503 14:13:41.233860 32201 raft_consensus.cc:348] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.234045 32201 raft_consensus.cc:374] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.234113 32201 raft_consensus.cc:729] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.234341 32201 consensus_queue.cc:260] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.234786 32201 ts_tablet_manager.cc:1398] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:13:41.306707 32191 raft_consensus.cc:482] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.306903 32191 raft_consensus.cc:504] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.307615 32191 leader_election.cc:288] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:41.307991 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e62601bbbf4686a68333c4acd8cd05" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:41.308259 30939 raft_consensus.cc:2454] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:41.308262 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e62601bbbf4686a68333c4acd8cd05" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:41.308466 28304 raft_consensus.cc:2454] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:41.308732 31138 leader_election.cc:302] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:41.309041 32191 raft_consensus.cc:2781] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.309149 32191 raft_consensus.cc:482] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.309271 32191 raft_consensus.cc:3037] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.311523 32191 raft_consensus.cc:504] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.312232 32191 leader_election.cc:288] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:41.312546 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e62601bbbf4686a68333c4acd8cd05" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:41.312657 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2e62601bbbf4686a68333c4acd8cd05" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:41.312773 30939 raft_consensus.cc:3037] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.312842 28304 raft_consensus.cc:3037] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.315582 30939 raft_consensus.cc:2454] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:41.315598 28304 raft_consensus.cc:2454] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:41.315971 31138 leader_election.cc:302] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:41.316560 32191 raft_consensus.cc:2781] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.316732 32191 raft_consensus.cc:686] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:41.317224 32191 consensus_queue.cc:237] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:41.320583 27901 catalog_manager.cc:5484] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:41.331315 32190 consensus_queue.cc:1035] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.335789 32116 consensus_queue.cc:1035] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.337841 32082 consensus_queue.cc:1035] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.353642 32082 consensus_queue.cc:1035] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.431277 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:41.436538 28304 raft_consensus.cc:1261] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:41.437005 30939 raft_consensus.cc:1261] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:41.437178 32191 consensus_queue.cc:1035] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.437624 32193 consensus_queue.cc:1035] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.442364 27898 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet d2e62601bbbf4686a68333c4acd8cd05 (table rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3]). Expected version 1 got 0
I20240503 14:13:41.443101 32181 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:41.443552 32181 tablet.cc:1620] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:41.451066 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3] alter complete (version 1)
I20240503 14:13:41.454399 32212 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:41.455440 32212 tablet.cc:1620] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:41.455021 27898 catalog_manager.cc:5576] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889) does not have the latest schema for tablet d2e62601bbbf4686a68333c4acd8cd05 (table rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3]). Expected version 1 got 0
I20240503 14:13:41.461459 32214 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:41.462126 32214 tablet.cc:1620] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:41.462417 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:41.463681 27899 catalog_manager.cc:5576] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889) does not have the latest schema for tablet d2e62601bbbf4686a68333c4acd8cd05 (table rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3]). Expected version 1 got 0
I20240503 14:13:41.472036 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 9202075be4bc472ebe781f577bb50ba9 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.472381 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 9202075be4bc472ebe781f577bb50ba9 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.472486 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9202075be4bc472ebe781f577bb50ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.472776 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9202075be4bc472ebe781f577bb50ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.472693 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 45da3dab7a204d378273a8985d91ace8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.473088 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45da3dab7a204d378273a8985d91ace8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.476223 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 98d39010d3e24d9fbd33c559568f7330 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.476683 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d39010d3e24d9fbd33c559568f7330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.477424 32201 tablet_bootstrap.cc:492] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:41.478550 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 45da3dab7a204d378273a8985d91ace8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.478605 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 9202075be4bc472ebe781f577bb50ba9 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.478660 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 98d39010d3e24d9fbd33c559568f7330 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.478973 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45da3dab7a204d378273a8985d91ace8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.479022 32201 tablet_bootstrap.cc:654] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.479060 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9202075be4bc472ebe781f577bb50ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.482882 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d39010d3e24d9fbd33c559568f7330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.483294 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 98d39010d3e24d9fbd33c559568f7330 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.483712 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d39010d3e24d9fbd33c559568f7330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.483294 30788 tablet_service.cc:1467] Processing CreateTablet for tablet 45da3dab7a204d378273a8985d91ace8 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.485069 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45da3dab7a204d378273a8985d91ace8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.491511 32193 consensus_queue.cc:1035] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.497968 32168 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:41.499573 32168 tablet_bootstrap.cc:654] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.510572 32168 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:41.510766 32168 ts_tablet_manager.cc:1367] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240503 14:13:41.513171 32193 consensus_queue.cc:1035] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.515223 32168 raft_consensus.cc:348] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.515467 32168 raft_consensus.cc:374] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.515549 32168 raft_consensus.cc:729] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.515799 32168 consensus_queue.cc:260] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.517931 32219 tablet_bootstrap.cc:492] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:41.518746 32168 ts_tablet_manager.cc:1398] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:41.519027 32168 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:41.521131 32168 tablet_bootstrap.cc:654] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.524854 32201 tablet_bootstrap.cc:492] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:41.525122 32201 ts_tablet_manager.cc:1367] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.017s
I20240503 14:13:41.526316 32201 raft_consensus.cc:348] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.526628 32201 raft_consensus.cc:374] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.526769 32201 raft_consensus.cc:729] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.527845 32201 consensus_queue.cc:260] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.529381 32218 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:41.533160 32201 ts_tablet_manager.cc:1398] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 14:13:41.534606 32219 tablet_bootstrap.cc:654] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.540863 32218 tablet_bootstrap.cc:654] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.547884 32219 tablet_bootstrap.cc:492] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:41.548122 32219 ts_tablet_manager.cc:1367] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 14:13:41.549005 32168 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:41.549180 32168 ts_tablet_manager.cc:1367] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.004s
I20240503 14:13:41.549686 32219 raft_consensus.cc:348] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.549870 32168 raft_consensus.cc:348] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:41.550091 32168 raft_consensus.cc:374] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.550174 32168 raft_consensus.cc:729] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.550307 32219 raft_consensus.cc:374] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.550452 32219 raft_consensus.cc:729] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.550655 32219 consensus_queue.cc:260] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.550634 32168 consensus_queue.cc:260] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:41.550873 32191 raft_consensus.cc:482] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.551019 32191 raft_consensus.cc:504] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:41.551780 32191 leader_election.cc:288] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:41.553030 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da3dab7a204d378273a8985d91ace8" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
W20240503 14:13:41.554544 31136 leader_election.cc:341] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:41.557950 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:41.563246 32168 ts_tablet_manager.cc:1398] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:13:41.563730 32218 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:41.564852 32218 ts_tablet_manager.cc:1367] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.000s
I20240503 14:13:41.567580 32218 raft_consensus.cc:348] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.568610 32212 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:41.568727 32218 raft_consensus.cc:374] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.568861 32218 raft_consensus.cc:729] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.569119 32212 tablet.cc:1620] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:41.569164 32218 consensus_queue.cc:260] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.570142 32216 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:41.572299 32216 tablet_bootstrap.cc:654] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.572554 32219 ts_tablet_manager.cc:1398] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20240503 14:13:41.573001 32214 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:41.573473 32214 tablet.cc:1620] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:41.575670 32218 ts_tablet_manager.cc:1398] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:13:41.575973 32218 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:41.589385 31986 raft_consensus.cc:482] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.589584 31986 raft_consensus.cc:504] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.590562 31986 leader_election.cc:288] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:41.590725 32181 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:41.591262 32181 tablet.cc:1620] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:41.591182 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:41.591424 31071 raft_consensus.cc:2454] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:41.591962 28239 leader_election.cc:302] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:41.592856 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
W20240503 14:13:41.593575 28237 leader_election.cc:341] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:41.593999 31986 raft_consensus.cc:2781] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.594135 31986 raft_consensus.cc:482] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.594271 31986 raft_consensus.cc:3037] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.596022 32218 tablet_bootstrap.cc:654] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.597793 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=5c3c846c469443b1ada12705c8876fe3] alter complete (version 2)
I20240503 14:13:41.597910 31986 raft_consensus.cc:504] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.600770 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
W20240503 14:13:41.601682 28237 leader_election.cc:341] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:41.602566 31986 leader_election.cc:288] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:41.603242 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:41.603485 31071 raft_consensus.cc:3037] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.605516 31071 raft_consensus.cc:2454] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:41.605895 28239 leader_election.cc:302] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters: d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:13:41.606469 32125 raft_consensus.cc:2781] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.609395 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da3dab7a204d378273a8985d91ace8" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
W20240503 14:13:41.609926 31137 leader_election.cc:341] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:41.610071 31137 leader_election.cc:302] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:13:41.610893 32228 raft_consensus.cc:2726] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:41.615566 32218 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:41.615783 32218 ts_tablet_manager.cc:1367] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.003s
I20240503 14:13:41.616528 32218 raft_consensus.cc:348] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:41.617126 32218 raft_consensus.cc:374] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.617265 32218 raft_consensus.cc:729] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.617581 32218 consensus_queue.cc:260] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:41.619480 32218 ts_tablet_manager.cc:1398] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:41.626276 32216 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:41.626502 32216 ts_tablet_manager.cc:1367] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.003s
I20240503 14:13:41.627300 32216 raft_consensus.cc:348] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.627573 32216 raft_consensus.cc:374] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.627712 32216 raft_consensus.cc:729] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.628217 32216 consensus_queue.cc:260] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.629758 32216 ts_tablet_manager.cc:1398] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:41.630079 32216 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:41.631814 32216 tablet_bootstrap.cc:654] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.633816 32125 raft_consensus.cc:686] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:41.634560 32125 consensus_queue.cc:237] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.642100 32216 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:41.642293 32216 ts_tablet_manager.cc:1367] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 14:13:41.643097 32216 raft_consensus.cc:348] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:41.643383 32216 raft_consensus.cc:374] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.643512 32216 raft_consensus.cc:729] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.643846 32216 consensus_queue.cc:260] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:41.644584 32216 ts_tablet_manager.cc:1398] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:41.647495 32216 tablet_bootstrap.cc:492] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:41.651903 32216 tablet_bootstrap.cc:654] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.658568 32142 raft_consensus.cc:482] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.658730 32142 raft_consensus.cc:504] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.658896 27898 catalog_manager.cc:5484] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:41.659425 32142 leader_election.cc:288] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:41.660858 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d39010d3e24d9fbd33c559568f7330" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:41.661171 31201 raft_consensus.cc:2454] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:41.661325 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d39010d3e24d9fbd33c559568f7330" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:41.661585 30810 raft_consensus.cc:2454] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:41.662701 28107 leader_election.cc:302] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:41.663025 32234 raft_consensus.cc:2781] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.663158 32234 raft_consensus.cc:482] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.663292 32234 raft_consensus.cc:3037] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.666683 32234 raft_consensus.cc:504] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.668252 32216 tablet_bootstrap.cc:492] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:41.668493 32216 ts_tablet_manager.cc:1367] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.003s
I20240503 14:13:41.669289 32216 raft_consensus.cc:348] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.669623 32216 raft_consensus.cc:374] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.669764 32216 raft_consensus.cc:729] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.670022 32216 consensus_queue.cc:260] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.671800 32216 ts_tablet_manager.cc:1398] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:41.673177 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d39010d3e24d9fbd33c559568f7330" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:41.673400 30810 raft_consensus.cc:3037] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.676290 30810 raft_consensus.cc:2454] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:41.676673 28107 leader_election.cc:302] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:41.676985 32142 raft_consensus.cc:2781] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.677076 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d39010d3e24d9fbd33c559568f7330" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:41.677256 31203 raft_consensus.cc:3037] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.679940 31203 raft_consensus.cc:2454] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:41.682686 32234 leader_election.cc:288] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:41.683174 32142 raft_consensus.cc:686] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:41.683495 32142 consensus_queue.cc:237] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:41.687044 27898 catalog_manager.cc:5484] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:41.690282 32145 raft_consensus.cc:482] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.690465 32145 raft_consensus.cc:504] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:41.691123 32145 leader_election.cc:288] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:41.691663 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:41.691920 31072 raft_consensus.cc:2379] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0dbfb0e43f948c3922926aacc365fc1 in current term 1: Already voted for candidate a4cdf05256a0407c8929ac43eb55d44f in this term.
I20240503 14:13:41.691885 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:41.692605 30745 leader_election.cc:302] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:41.693403 32145 raft_consensus.cc:3037] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.695785 32145 raft_consensus.cc:2726] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:41.703229 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:41.709808 32212 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:41.710547 32214 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:41.713716 32181 tablet.cc:1722] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:41.717564 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=5c3c846c469443b1ada12705c8876fe3] alter complete (version 3)
I20240503 14:13:41.821033 27901 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:41.821326 27901 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:41.824187 27901 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet d2e62601bbbf4686a68333c4acd8cd05
I20240503 14:13:41.825096 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e62601bbbf4686a68333c4acd8cd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:41.825085 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e62601bbbf4686a68333c4acd8cd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:41.825126 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet d2e62601bbbf4686a68333c4acd8cd05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:41.825736 32242 tablet_replica.cc:330] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:41.825943 32242 raft_consensus.cc:2229] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:41.826290 32242 raft_consensus.cc:2258] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.826757 32241 tablet_replica.cc:330] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:41.826951 32241 raft_consensus.cc:2229] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.827152 32241 raft_consensus.cc:2258] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.827472 32242 ts_tablet_manager.cc:1875] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.827709 32240 tablet_replica.cc:330] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:41.828311 32241 ts_tablet_manager.cc:1875] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.828593 32240 raft_consensus.cc:2229] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.828840 32240 raft_consensus.cc:2258] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.829879 32240 ts_tablet_manager.cc:1875] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.833019 32242 ts_tablet_manager.cc:1888] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.833145 32242 log.cc:1196] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/d2e62601bbbf4686a68333c4acd8cd05
I20240503 14:13:41.833622 32242 ts_tablet_manager.cc:1909] T d2e62601bbbf4686a68333c4acd8cd05 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:41.834707 32241 ts_tablet_manager.cc:1888] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.834864 32241 log.cc:1196] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/d2e62601bbbf4686a68333c4acd8cd05
I20240503 14:13:41.834805 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet d2e62601bbbf4686a68333c4acd8cd05 (table rebalancer_dynamic_table_new_name [id=5c3c846c469443b1ada12705c8876fe3]) successfully deleted
I20240503 14:13:41.835413 32241 ts_tablet_manager.cc:1909] T d2e62601bbbf4686a68333c4acd8cd05 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:41.836437 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet d2e62601bbbf4686a68333c4acd8cd05 (table rebalancer_dynamic_table_new_name [id=5c3c846c469443b1ada12705c8876fe3]) successfully deleted
I20240503 14:13:41.837834 32240 ts_tablet_manager.cc:1888] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.837932 32240 log.cc:1196] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/d2e62601bbbf4686a68333c4acd8cd05
I20240503 14:13:41.838260 32240 ts_tablet_manager.cc:1909] T d2e62601bbbf4686a68333c4acd8cd05 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:41.839092 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet d2e62601bbbf4686a68333c4acd8cd05 (table rebalancer_dynamic_table_new_name [id=5c3c846c469443b1ada12705c8876fe3]) successfully deleted
I20240503 14:13:41.926427 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:41.934074 31183 tablet_service.cc:1467] Processing CreateTablet for tablet a9340c831f4b4554977436c3ac8d76e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b]), partition=
I20240503 14:13:41.935106 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9340c831f4b4554977436c3ac8d76e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.935415 28414 tablet_service.cc:1467] Processing CreateTablet for tablet a9340c831f4b4554977436c3ac8d76e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b]), partition=
I20240503 14:13:41.935420 31051 tablet_service.cc:1467] Processing CreateTablet for tablet a9340c831f4b4554977436c3ac8d76e6 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b]), partition=
I20240503 14:13:41.936143 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9340c831f4b4554977436c3ac8d76e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.936268 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9340c831f4b4554977436c3ac8d76e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.943675 32168 tablet_bootstrap.cc:492] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:41.944955 32168 tablet_bootstrap.cc:654] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.947602 32168 tablet_bootstrap.cc:492] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:41.947796 32168 ts_tablet_manager.cc:1367] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:41.948561 32219 tablet_bootstrap.cc:492] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:41.948740 32168 raft_consensus.cc:348] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:41.948953 32168 raft_consensus.cc:374] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.949048 32168 raft_consensus.cc:729] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.949297 32168 consensus_queue.cc:260] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:41.949224 32247 tablet_bootstrap.cc:492] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:41.949795 32168 ts_tablet_manager.cc:1398] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:41.950263 32219 tablet_bootstrap.cc:654] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.951501 32247 tablet_bootstrap.cc:654] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.954556 32247 tablet_bootstrap.cc:492] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:41.954665 32219 tablet_bootstrap.cc:492] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:41.954705 32247 ts_tablet_manager.cc:1367] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 14:13:41.954830 32219 ts_tablet_manager.cc:1367] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20240503 14:13:41.955428 32219 raft_consensus.cc:348] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:41.955493 32247 raft_consensus.cc:348] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:41.955649 32219 raft_consensus.cc:374] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.955727 32219 raft_consensus.cc:729] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.955765 32247 raft_consensus.cc:374] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.955891 32247 raft_consensus.cc:729] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.955981 32219 consensus_queue.cc:260] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:41.956156 32247 consensus_queue.cc:260] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:41.956575 32219 ts_tablet_manager.cc:1398] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:41.957017 32247 ts_tablet_manager.cc:1398] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:42.015601 32238 raft_consensus.cc:482] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.015791 32238 raft_consensus.cc:504] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.016520 32238 leader_election.cc:288] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:42.016921 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da3dab7a204d378273a8985d91ace8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:42.016992 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da3dab7a204d378273a8985d91ace8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:42.017146 31203 raft_consensus.cc:2454] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:42.017242 30810 raft_consensus.cc:2454] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:42.017609 28108 leader_election.cc:302] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:42.018014 32238 raft_consensus.cc:2781] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.018136 32238 raft_consensus.cc:482] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.018252 32238 raft_consensus.cc:3037] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.020637 32238 raft_consensus.cc:504] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.021628 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da3dab7a204d378273a8985d91ace8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:42.021790 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45da3dab7a204d378273a8985d91ace8" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:42.021914 31203 raft_consensus.cc:3037] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.021986 30810 raft_consensus.cc:3037] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.022078 32238 leader_election.cc:288] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:42.024044 30810 raft_consensus.cc:2454] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:42.024396 28107 leader_election.cc:302] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:42.024751 32142 raft_consensus.cc:2781] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.024883 32142 raft_consensus.cc:686] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:42.024950 31203 raft_consensus.cc:2454] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:42.025135 32142 consensus_queue.cc:237] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.028501 27901 catalog_manager.cc:5484] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:42.097383 32238 consensus_queue.cc:1035] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.111753 32142 consensus_queue.cc:1035] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.118268 32233 consensus_queue.cc:1035] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.122292 32152 raft_consensus.cc:482] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.122499 32152 raft_consensus.cc:504] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.122728 32125 consensus_queue.cc:1035] T 9202075be4bc472ebe781f577bb50ba9 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.123204 32152 leader_election.cc:288] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:42.124089 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9340c831f4b4554977436c3ac8d76e6" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:42.124362 28434 raft_consensus.cc:2454] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:42.125013 31008 leader_election.cc:302] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:42.125351 32152 raft_consensus.cc:2781] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.125478 32152 raft_consensus.cc:482] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.125613 32152 raft_consensus.cc:3037] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.125593 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9340c831f4b4554977436c3ac8d76e6" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:42.125900 31203 raft_consensus.cc:2454] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:42.127969 32152 raft_consensus.cc:504] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.128582 32152 leader_election.cc:288] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:42.129035 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9340c831f4b4554977436c3ac8d76e6" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:42.129261 28434 raft_consensus.cc:3037] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.130185 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9340c831f4b4554977436c3ac8d76e6" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:42.130402 31203 raft_consensus.cc:3037] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.131837 28434 raft_consensus.cc:2454] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:42.132251 31008 leader_election.cc:302] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:42.132539 31967 raft_consensus.cc:2781] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.132612 31203 raft_consensus.cc:2454] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:42.132683 31967 raft_consensus.cc:686] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:42.132918 31967 consensus_queue.cc:237] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.136309 27901 catalog_manager.cc:5484] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:42.276460 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:42.282493 31203 raft_consensus.cc:1261] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:42.282734 28434 raft_consensus.cc:1261] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:42.283232 31967 consensus_queue.cc:1035] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.283687 32152 consensus_queue.cc:1035] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.288827 27899 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet a9340c831f4b4554977436c3ac8d76e6 (table rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b]). Expected version 1 got 0
I20240503 14:13:42.290071 32263 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:42.290158 27901 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet a9340c831f4b4554977436c3ac8d76e6 (table rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b]). Expected version 1 got 0
I20240503 14:13:42.290674 32263 tablet.cc:1620] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:42.296718 32264 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:42.298040 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b] alter complete (version 1)
I20240503 14:13:42.298640 32262 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:42.299283 32262 tablet.cc:1620] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:42.299580 27899 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet a9340c831f4b4554977436c3ac8d76e6 (table rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b]). Expected version 1 got 0
I20240503 14:13:42.301272 32264 tablet.cc:1620] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:42.402423 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:42.409801 32262 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:42.410491 32262 tablet.cc:1620] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:42.410785 32264 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:42.411084 32263 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:42.411284 32264 tablet.cc:1620] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:42.411531 32263 tablet.cc:1620] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:42.421326 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=04127995ffcb4e08a5373da64bd62a5b] alter complete (version 2)
I20240503 14:13:42.471237 32234 consensus_queue.cc:1035] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.475443 32234 consensus_queue.cc:1035] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.525532 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:42.532801 32262 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:42.533488 32263 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:42.534287 32264 tablet.cc:1722] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:42.545281 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=04127995ffcb4e08a5373da64bd62a5b] alter complete (version 3)
I20240503 14:13:42.587684 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:42.596663 30919 tablet_service.cc:1467] Processing CreateTablet for tablet fba8583c4254431e8f95633c4998fc88 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.597046 28023 tablet_service.cc:1467] Processing CreateTablet for tablet fba8583c4254431e8f95633c4998fc88 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.597165 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba8583c4254431e8f95633c4998fc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.597443 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba8583c4254431e8f95633c4998fc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.597819 31051 tablet_service.cc:1467] Processing CreateTablet for tablet a88d8b316d064c4e8fdccaba77d656c2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.598213 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88d8b316d064c4e8fdccaba77d656c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.601538 32269 tablet_bootstrap.cc:492] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:42.602675 28414 tablet_service.cc:1467] Processing CreateTablet for tablet fba8583c4254431e8f95633c4998fc88 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.603085 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fba8583c4254431e8f95633c4998fc88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.604127 30919 tablet_service.cc:1467] Processing CreateTablet for tablet a88d8b316d064c4e8fdccaba77d656c2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.604547 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88d8b316d064c4e8fdccaba77d656c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.604598 30920 tablet_service.cc:1467] Processing CreateTablet for tablet f6151ea340334090a5c0e79ca65af5c1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.605849 30789 tablet_service.cc:1467] Processing CreateTablet for tablet f6151ea340334090a5c0e79ca65af5c1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.606559 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6151ea340334090a5c0e79ca65af5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.606632 28022 tablet_service.cc:1467] Processing CreateTablet for tablet a88d8b316d064c4e8fdccaba77d656c2 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.607009 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88d8b316d064c4e8fdccaba77d656c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.607594 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6151ea340334090a5c0e79ca65af5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.607813 28282 tablet_service.cc:1467] Processing CreateTablet for tablet f6151ea340334090a5c0e79ca65af5c1 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.608206 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6151ea340334090a5c0e79ca65af5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.626576 32269 tablet_bootstrap.cc:654] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.633430 32272 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:42.634482 32269 tablet_bootstrap.cc:492] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:42.634742 32269 ts_tablet_manager.cc:1367] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240503 14:13:42.635915 32271 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:42.636732 32269 raft_consensus.cc:348] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.637516 32272 tablet_bootstrap.cc:654] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.639030 32274 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:42.639186 32269 raft_consensus.cc:374] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.639339 32269 raft_consensus.cc:729] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.639632 32269 consensus_queue.cc:260] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.641887 32274 tablet_bootstrap.cc:654] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.642590 32269 ts_tablet_manager.cc:1398] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:42.643690 32271 tablet_bootstrap.cc:654] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.644008 32269 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:42.649484 27901 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:42.649740 27901 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:42.650130 32273 tablet_bootstrap.cc:492] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:42.650298 32269 tablet_bootstrap.cc:654] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.652199 32273 tablet_bootstrap.cc:654] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.652400 32271 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:42.652544 32271 ts_tablet_manager.cc:1367] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 14:13:42.653260 32271 raft_consensus.cc:348] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.654402 32271 raft_consensus.cc:374] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.654572 32271 raft_consensus.cc:729] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.654881 32271 consensus_queue.cc:260] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.656811 32270 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:42.657362 32271 ts_tablet_manager.cc:1398] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:42.658300 27901 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet a9340c831f4b4554977436c3ac8d76e6
I20240503 14:13:42.660157 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet a9340c831f4b4554977436c3ac8d76e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:42 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:42.660522 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet a9340c831f4b4554977436c3ac8d76e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:42 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:42.671535 32269 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:42.671723 32269 ts_tablet_manager.cc:1367] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20240503 14:13:42.672544 32269 raft_consensus.cc:348] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.672686 32272 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:42.672791 32269 raft_consensus.cc:374] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.672891 32272 ts_tablet_manager.cc:1367] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.009s
I20240503 14:13:42.673931 32272 raft_consensus.cc:348] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.674332 32272 raft_consensus.cc:374] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.674465 32272 raft_consensus.cc:729] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.674746 32272 consensus_queue.cc:260] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.675397 32272 ts_tablet_manager.cc:1398] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:42.675683 32272 tablet_bootstrap.cc:492] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:42.678251 32272 tablet_bootstrap.cc:654] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.678510 32282 tablet_replica.cc:330] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:42.679406 32269 raft_consensus.cc:729] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.679869 32282 raft_consensus.cc:2229] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:42.680418 32269 consensus_queue.cc:260] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.681596 32272 tablet_bootstrap.cc:492] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:42.681838 32282 raft_consensus.cc:2258] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.681957 32272 ts_tablet_manager.cc:1367] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:42.683210 32269 ts_tablet_manager.cc:1398] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 14:13:42.683122 32272 raft_consensus.cc:348] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.683332 32272 raft_consensus.cc:374] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.683475 32272 raft_consensus.cc:729] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.683836 32272 consensus_queue.cc:260] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.684506 32269 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:42.684573 32272 ts_tablet_manager.cc:1398] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:42.685320 32282 ts_tablet_manager.cc:1875] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:42.688125 32281 tablet_replica.cc:330] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:42.688395 32281 raft_consensus.cc:2229] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:42.688632 32281 raft_consensus.cc:2258] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.660286 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet a9340c831f4b4554977436c3ac8d76e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:42 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:42.691612 32281 ts_tablet_manager.cc:1875] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:42.696525 32281 ts_tablet_manager.cc:1888] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:42.698683 32270 tablet_bootstrap.cc:654] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.698912 32269 tablet_bootstrap.cc:654] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.699542 32274 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:42.699784 32274 ts_tablet_manager.cc:1367] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.000s
I20240503 14:13:42.700601 32274 raft_consensus.cc:348] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.701011 32274 raft_consensus.cc:374] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.701156 32274 raft_consensus.cc:729] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.701479 32274 consensus_queue.cc:260] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.702311 32274 ts_tablet_manager.cc:1398] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:42.703914 32282 ts_tablet_manager.cc:1888] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:42.704368 32282 log.cc:1196] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/a9340c831f4b4554977436c3ac8d76e6
I20240503 14:13:42.705706 32282 ts_tablet_manager.cc:1909] T a9340c831f4b4554977436c3ac8d76e6 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:42.706732 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet a9340c831f4b4554977436c3ac8d76e6 (table rebalancer_dynamic_table_new_name [id=04127995ffcb4e08a5373da64bd62a5b]) successfully deleted
I20240503 14:13:42.707798 32285 tablet_replica.cc:330] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:42.708148 32285 raft_consensus.cc:2229] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:42.708701 32285 raft_consensus.cc:2258] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.709734 32285 ts_tablet_manager.cc:1875] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:42.705104 32273 tablet_bootstrap.cc:492] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:42.714426 32273 ts_tablet_manager.cc:1367] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.000s
I20240503 14:13:42.715286 32273 raft_consensus.cc:348] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.715483 32273 raft_consensus.cc:374] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.715572 32273 raft_consensus.cc:729] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.715814 32273 consensus_queue.cc:260] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.716481 32273 ts_tablet_manager.cc:1398] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:42.716693 32281 log.cc:1196] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9340c831f4b4554977436c3ac8d76e6
I20240503 14:13:42.717093 32269 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:42.717334 32281 ts_tablet_manager.cc:1909] T a9340c831f4b4554977436c3ac8d76e6 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:42.718000 32285 ts_tablet_manager.cc:1888] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:42.718127 32285 log.cc:1196] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/a9340c831f4b4554977436c3ac8d76e6
I20240503 14:13:42.717286 32269 ts_tablet_manager.cc:1367] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.002s
I20240503 14:13:42.718616 32285 ts_tablet_manager.cc:1909] T a9340c831f4b4554977436c3ac8d76e6 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:42.719535 32270 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:42.719727 32270 ts_tablet_manager.cc:1367] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.000s
I20240503 14:13:42.720469 32270 raft_consensus.cc:348] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.720782 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet a9340c831f4b4554977436c3ac8d76e6 (table rebalancer_dynamic_table_new_name [id=04127995ffcb4e08a5373da64bd62a5b]) successfully deleted
I20240503 14:13:42.721347 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet a9340c831f4b4554977436c3ac8d76e6 (table rebalancer_dynamic_table_new_name [id=04127995ffcb4e08a5373da64bd62a5b]) successfully deleted
I20240503 14:13:42.721493 32269 raft_consensus.cc:348] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.721688 32269 raft_consensus.cc:374] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.721781 32269 raft_consensus.cc:729] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.721951 32270 raft_consensus.cc:374] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.722102 32270 raft_consensus.cc:729] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.722389 32270 consensus_queue.cc:260] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.723146 32270 ts_tablet_manager.cc:1398] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:42.723363 32269 consensus_queue.cc:260] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.724105 32269 ts_tablet_manager.cc:1398] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:13:42.761330 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:42.769214 30789 tablet_service.cc:1467] Processing CreateTablet for tablet a4eb0d84c61642d9b73cf479d94f6498 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb]), partition=
I20240503 14:13:42.769284 30919 tablet_service.cc:1467] Processing CreateTablet for tablet a4eb0d84c61642d9b73cf479d94f6498 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb]), partition=
I20240503 14:13:42.769371 31051 tablet_service.cc:1467] Processing CreateTablet for tablet a4eb0d84c61642d9b73cf479d94f6498 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb]), partition=
I20240503 14:13:42.769635 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4eb0d84c61642d9b73cf479d94f6498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.769635 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4eb0d84c61642d9b73cf479d94f6498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.769747 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4eb0d84c61642d9b73cf479d94f6498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.776463 32190 raft_consensus.cc:482] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.776659 32190 raft_consensus.cc:504] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.777626 32190 leader_election.cc:288] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:42.777990 32269 tablet_bootstrap.cc:492] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:42.778488 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8583c4254431e8f95633c4998fc88" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:42.778748 32270 tablet_bootstrap.cc:492] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:42.778744 28042 raft_consensus.cc:2454] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:42.779186 30877 leader_election.cc:302] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:42.779253 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8583c4254431e8f95633c4998fc88" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:42.779477 28432 raft_consensus.cc:2454] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:42.779536 32190 raft_consensus.cc:2781] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.779628 32190 raft_consensus.cc:482] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.779657 32269 tablet_bootstrap.cc:654] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.779745 32190 raft_consensus.cc:3037] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.780293 32270 tablet_bootstrap.cc:654] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.780997 32271 tablet_bootstrap.cc:492] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:42.782555 32271 tablet_bootstrap.cc:654] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.782915 32190 raft_consensus.cc:504] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.784186 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8583c4254431e8f95633c4998fc88" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:42.784417 28042 raft_consensus.cc:3037] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.784456 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8583c4254431e8f95633c4998fc88" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:42.784652 28432 raft_consensus.cc:3037] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.785571 32270 tablet_bootstrap.cc:492] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:42.785688 32270 ts_tablet_manager.cc:1367] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240503 14:13:42.786458 32270 raft_consensus.cc:348] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.787257 32269 tablet_bootstrap.cc:492] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:42.787315 28432 raft_consensus.cc:2454] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:42.787315 28042 raft_consensus.cc:2454] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:42.787420 32269 ts_tablet_manager.cc:1367] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:42.787773 30877 leader_election.cc:302] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:42.788079 32269 raft_consensus.cc:348] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.788187 32245 raft_consensus.cc:2781] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.788288 32269 raft_consensus.cc:374] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.788379 32269 raft_consensus.cc:729] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.788619 32269 consensus_queue.cc:260] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.788801 32245 raft_consensus.cc:686] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:42.789130 32245 consensus_queue.cc:237] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:42.789541 32271 tablet_bootstrap.cc:492] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:42.789709 32271 ts_tablet_manager.cc:1367] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:42.790488 32271 raft_consensus.cc:348] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.790721 32271 raft_consensus.cc:374] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.790824 32271 raft_consensus.cc:729] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.790896 32190 leader_election.cc:288] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:42.791041 32271 consensus_queue.cc:260] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.791535 32271 ts_tablet_manager.cc:1398] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:42.792312 32269 ts_tablet_manager.cc:1398] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:42.792510 32270 raft_consensus.cc:374] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.792703 32270 raft_consensus.cc:729] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.793092 32270 consensus_queue.cc:260] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.793912 27898 catalog_manager.cc:5484] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:42.796025 32270 ts_tablet_manager.cc:1398] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.010s user 0.001s sys 0.002s
I20240503 14:13:42.857396 32292 raft_consensus.cc:482] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.857602 32292 raft_consensus.cc:504] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.858798 32292 leader_election.cc:288] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:42.858896 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4eb0d84c61642d9b73cf479d94f6498" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:42.859095 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4eb0d84c61642d9b73cf479d94f6498" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:42.859163 31071 raft_consensus.cc:2454] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:42.859333 30809 raft_consensus.cc:2454] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:42.859663 30876 leader_election.cc:302] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:42.859990 32292 raft_consensus.cc:2781] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.860119 32292 raft_consensus.cc:482] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.860215 32292 raft_consensus.cc:3037] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.863788 32292 raft_consensus.cc:504] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.865305 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4eb0d84c61642d9b73cf479d94f6498" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:42.865545 31071 raft_consensus.cc:3037] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.865562 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4eb0d84c61642d9b73cf479d94f6498" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:42.865636 32292 leader_election.cc:288] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:42.865751 30809 raft_consensus.cc:3037] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.868577 30809 raft_consensus.cc:2454] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:42.869171 30874 leader_election.cc:302] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:42.870000 32292 raft_consensus.cc:2781] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.870628 31071 raft_consensus.cc:2454] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:42.870724 32292 raft_consensus.cc:686] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:42.871068 32292 consensus_queue.cc:237] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:42.885020 31967 raft_consensus.cc:482] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.885215 31967 raft_consensus.cc:504] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.886082 31967 leader_election.cc:288] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:42.888577 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88d8b316d064c4e8fdccaba77d656c2" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:42.888816 30939 raft_consensus.cc:2454] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:42.889393 27901 catalog_manager.cc:5484] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:42.889916 31007 leader_election.cc:302] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:42.893394 31967 raft_consensus.cc:2781] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.893520 31967 raft_consensus.cc:482] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.893621 31967 raft_consensus.cc:3037] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.895828 31967 raft_consensus.cc:504] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.896415 31967 leader_election.cc:288] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:42.896834 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88d8b316d064c4e8fdccaba77d656c2" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:42.897073 30939 raft_consensus.cc:3037] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.899855 30939 raft_consensus.cc:2454] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:42.900277 31007 leader_election.cc:302] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:42.901686 31967 raft_consensus.cc:2781] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.902374 31967 raft_consensus.cc:686] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:42.902681 31967 consensus_queue.cc:237] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.910215 27901 catalog_manager.cc:5484] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:42.913970 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88d8b316d064c4e8fdccaba77d656c2" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:42.913950 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a88d8b316d064c4e8fdccaba77d656c2" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:42.914260 28040 raft_consensus.cc:3037] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.917057 28040 raft_consensus.cc:2454] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:42.998800 32265 raft_consensus.cc:482] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.999002 32265 raft_consensus.cc:504] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:42.999801 32265 leader_election.cc:288] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:43.000161 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6151ea340334090a5c0e79ca65af5c1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:43.000411 28303 raft_consensus.cc:2454] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:43.000880 30746 leader_election.cc:302] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:43.000916 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6151ea340334090a5c0e79ca65af5c1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:43.001159 30939 raft_consensus.cc:2454] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:43.001564 32265 raft_consensus.cc:2781] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.001669 32265 raft_consensus.cc:482] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.001740 32265 raft_consensus.cc:3037] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.003741 32265 raft_consensus.cc:504] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:43.004778 32265 leader_election.cc:288] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:43.005128 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6151ea340334090a5c0e79ca65af5c1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:43.005239 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6151ea340334090a5c0e79ca65af5c1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:43.005349 28303 raft_consensus.cc:3037] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.005448 30939 raft_consensus.cc:3037] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.006048 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:43.008337 28303 raft_consensus.cc:2454] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:43.008572 30939 raft_consensus.cc:2454] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:43.008872 30746 leader_election.cc:302] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:43.009347 32265 raft_consensus.cc:2781] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.009496 32265 raft_consensus.cc:686] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:43.009713 32265 consensus_queue.cc:237] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:43.014989 31071 raft_consensus.cc:1261] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.015282 30809 raft_consensus.cc:1261] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.015596 27901 catalog_manager.cc:5484] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:43.015830 32190 consensus_queue.cc:1035] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.016049 32292 consensus_queue.cc:1035] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.020555 32304 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.020738 32262 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.020823 27901 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet a4eb0d84c61642d9b73cf479d94f6498 (table rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb]). Expected version 1 got 0
I20240503 14:13:43.021140 32304 tablet.cc:1620] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.021169 32262 tablet.cc:1620] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.029654 27900 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet a4eb0d84c61642d9b73cf479d94f6498 (table rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb]). Expected version 1 got 0
I20240503 14:13:43.031309 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb] alter complete (version 1)
I20240503 14:13:43.033084 32306 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.041388 27900 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet a4eb0d84c61642d9b73cf479d94f6498 (table rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb]). Expected version 1 got 0
I20240503 14:13:43.041929 32306 tablet.cc:1620] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:43.140262 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:43.147727 32306 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.147802 32304 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.148391 32306 tablet.cc:1620] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:43.148321 32262 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.148524 32304 tablet.cc:1620] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.148769 32262 tablet.cc:1620] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.164358 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=d858518a17a94ff89a830eb0f22c88bb] alter complete (version 2)
I20240503 14:13:43.269061 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:43.277141 32306 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:43.278738 32262 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:43.278851 32304 tablet.cc:1722] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:43.286523 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=d858518a17a94ff89a830eb0f22c88bb] alter complete (version 3)
I20240503 14:13:43.390846 27898 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:43.391065 27898 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:43.393992 27898 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet a4eb0d84c61642d9b73cf479d94f6498
I20240503 14:13:43.394794 30919 tablet_service.cc:1514] Processing DeleteTablet for tablet a4eb0d84c61642d9b73cf479d94f6498 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:43 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:43.394912 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet a4eb0d84c61642d9b73cf479d94f6498 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:43 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:43.395104 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet a4eb0d84c61642d9b73cf479d94f6498 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:43 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:43.395406 32308 tablet_replica.cc:330] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:43.395638 32308 raft_consensus.cc:2229] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:43.396044 32308 raft_consensus.cc:2258] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.396533 32310 tablet_replica.cc:330] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:43.396811 32309 tablet_replica.cc:330] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:43.396811 32310 raft_consensus.cc:2229] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:43.397025 32310 raft_consensus.cc:2258] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.397038 32309 raft_consensus.cc:2229] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:43.397074 32308 ts_tablet_manager.cc:1875] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:43.397286 32309 raft_consensus.cc:2258] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.397825 32310 ts_tablet_manager.cc:1875] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:43.398124 32309 ts_tablet_manager.cc:1875] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:43.402484 32309 ts_tablet_manager.cc:1888] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:43.402595 32309 log.cc:1196] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/a4eb0d84c61642d9b73cf479d94f6498
I20240503 14:13:43.403015 32309 ts_tablet_manager.cc:1909] T a4eb0d84c61642d9b73cf479d94f6498 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:43.403291 32308 ts_tablet_manager.cc:1888] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:43.403445 32308 log.cc:1196] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/a4eb0d84c61642d9b73cf479d94f6498
I20240503 14:13:43.403800 32310 ts_tablet_manager.cc:1888] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:43.403878 32308 ts_tablet_manager.cc:1909] T a4eb0d84c61642d9b73cf479d94f6498 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:43.403952 32310 log.cc:1196] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/a4eb0d84c61642d9b73cf479d94f6498
I20240503 14:13:43.404055 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet a4eb0d84c61642d9b73cf479d94f6498 (table rebalancer_dynamic_table_new_name [id=d858518a17a94ff89a830eb0f22c88bb]) successfully deleted
I20240503 14:13:43.404428 32310 ts_tablet_manager.cc:1909] T a4eb0d84c61642d9b73cf479d94f6498 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:43.404879 32190 consensus_queue.cc:1035] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.404946 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet a4eb0d84c61642d9b73cf479d94f6498 (table rebalancer_dynamic_table_new_name [id=d858518a17a94ff89a830eb0f22c88bb]) successfully deleted
I20240503 14:13:43.405455 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet a4eb0d84c61642d9b73cf479d94f6498 (table rebalancer_dynamic_table_new_name [id=d858518a17a94ff89a830eb0f22c88bb]) successfully deleted
I20240503 14:13:43.407342 32190 consensus_queue.cc:1035] T fba8583c4254431e8f95633c4998fc88 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.429240 32268 consensus_queue.cc:1035] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.434711 32268 consensus_queue.cc:1035] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.496241 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:43.504952 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 7a648901743a4d109bfb2309afc4f4ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31]), partition=
I20240503 14:13:43.504952 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 7a648901743a4d109bfb2309afc4f4ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31]), partition=
I20240503 14:13:43.505121 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 7a648901743a4d109bfb2309afc4f4ca (DEFAULT_TABLE table=rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31]), partition=
I20240503 14:13:43.505340 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a648901743a4d109bfb2309afc4f4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.505339 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a648901743a4d109bfb2309afc4f4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.505466 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a648901743a4d109bfb2309afc4f4ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.522303 32320 tablet_bootstrap.cc:492] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:43.522923 32318 tablet_bootstrap.cc:492] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:43.523602 32319 tablet_bootstrap.cc:492] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:43.524487 32320 tablet_bootstrap.cc:654] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.524709 32318 tablet_bootstrap.cc:654] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.525300 32319 tablet_bootstrap.cc:654] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.527114 32318 tablet_bootstrap.cc:492] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:43.527282 32318 ts_tablet_manager.cc:1367] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:43.527835 32318 raft_consensus.cc:348] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.528023 32318 raft_consensus.cc:374] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.528120 32318 raft_consensus.cc:729] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.528338 32318 consensus_queue.cc:260] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.528831 32318 ts_tablet_manager.cc:1398] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:43.529477 32319 tablet_bootstrap.cc:492] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:43.529655 32319 ts_tablet_manager.cc:1367] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240503 14:13:43.530427 32319 raft_consensus.cc:348] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.530695 32319 raft_consensus.cc:374] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.530833 32319 raft_consensus.cc:729] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.531384 32319 consensus_queue.cc:260] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.533907 32320 tablet_bootstrap.cc:492] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:43.534056 32320 ts_tablet_manager.cc:1367] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 14:13:43.534953 32320 raft_consensus.cc:348] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.535166 32320 raft_consensus.cc:374] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.535310 32320 raft_consensus.cc:729] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.535615 32320 consensus_queue.cc:260] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.535921 32299 consensus_queue.cc:1035] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.536237 32320 ts_tablet_manager.cc:1398] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:43.537130 32319 ts_tablet_manager.cc:1398] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:43.542485 32299 consensus_queue.cc:1035] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.579712 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:43.591447 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 9132d0f9d1ea4be49458a085c9e98a65 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.591758 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 9132d0f9d1ea4be49458a085c9e98a65 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.591835 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 9132d0f9d1ea4be49458a085c9e98a65 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.591989 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9132d0f9d1ea4be49458a085c9e98a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.592203 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9132d0f9d1ea4be49458a085c9e98a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.592223 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9132d0f9d1ea4be49458a085c9e98a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.595362 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 3a82a00c33e1412f93ab071d5b0d824a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.595364 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 611bac13cf32492b87c2f57752c59462 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.595753 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a82a00c33e1412f93ab071d5b0d824a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.595772 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611bac13cf32492b87c2f57752c59462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.595824 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 611bac13cf32492b87c2f57752c59462 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.596230 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611bac13cf32492b87c2f57752c59462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.600747 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 3a82a00c33e1412f93ab071d5b0d824a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.601145 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a82a00c33e1412f93ab071d5b0d824a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.602486 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 3a82a00c33e1412f93ab071d5b0d824a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.602895 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a82a00c33e1412f93ab071d5b0d824a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.602785 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 611bac13cf32492b87c2f57752c59462 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.603261 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 611bac13cf32492b87c2f57752c59462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.605762 32320 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:43.605863 32319 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:43.607364 32320 tablet_bootstrap.cc:654] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.617390 32320 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:43.617640 32320 ts_tablet_manager.cc:1367] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240503 14:13:43.618527 32320 raft_consensus.cc:348] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.618829 32320 raft_consensus.cc:374] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.618957 32320 raft_consensus.cc:729] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.619336 32320 consensus_queue.cc:260] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.621459 32319 tablet_bootstrap.cc:654] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.621874 32320 ts_tablet_manager.cc:1398] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:43.622182 32320 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:43.624001 32320 tablet_bootstrap.cc:654] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.633642 32319 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:43.633913 32319 ts_tablet_manager.cc:1367] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240503 14:13:43.634821 32319 raft_consensus.cc:348] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.635143 32319 raft_consensus.cc:374] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.635270 32319 raft_consensus.cc:729] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.635581 32319 consensus_queue.cc:260] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.640101 32319 ts_tablet_manager.cc:1398] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:43.646394 32234 raft_consensus.cc:482] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.646595 32234 raft_consensus.cc:504] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.646996 32332 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:43.647276 32234 leader_election.cc:288] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:43.647979 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a648901743a4d109bfb2309afc4f4ca" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:43.648278 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a648901743a4d109bfb2309afc4f4ca" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:43.648357 30939 raft_consensus.cc:2454] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:43.648516 31071 raft_consensus.cc:2454] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:43.648919 28109 leader_election.cc:302] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:43.649466 32234 raft_consensus.cc:2781] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.649569 32234 raft_consensus.cc:482] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.649685 32234 raft_consensus.cc:3037] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.652555 32234 raft_consensus.cc:504] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.653859 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a648901743a4d109bfb2309afc4f4ca" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:43.654335 30939 raft_consensus.cc:3037] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.655815 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a648901743a4d109bfb2309afc4f4ca" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:43.656049 31071 raft_consensus.cc:3037] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.660705 31071 raft_consensus.cc:2454] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:43.661118 28109 leader_election.cc:302] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:43.661909 32234 leader_election.cc:288] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:43.663452 32234 raft_consensus.cc:2781] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.663604 32234 raft_consensus.cc:686] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:43.663347 30939 raft_consensus.cc:2454] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:43.663843 32234 consensus_queue.cc:237] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:43.666896 32332 tablet_bootstrap.cc:654] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.674787 32332 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:43.676038 32320 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:43.676381 32332 ts_tablet_manager.cc:1367] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240503 14:13:43.677625 32332 raft_consensus.cc:348] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.679601 32320 ts_tablet_manager.cc:1367] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.002s
I20240503 14:13:43.680334 32332 raft_consensus.cc:374] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.680815 32332 raft_consensus.cc:729] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.680764 32320 raft_consensus.cc:348] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.681094 32320 raft_consensus.cc:374] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.681245 32320 raft_consensus.cc:729] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.681483 32332 consensus_queue.cc:260] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.681605 32320 consensus_queue.cc:260] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.683550 32332 ts_tablet_manager.cc:1398] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:43.684149 32320 ts_tablet_manager.cc:1398] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:43.689246 32332 tablet_bootstrap.cc:492] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:43.690981 32333 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:43.696979 32334 tablet_bootstrap.cc:492] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:43.697774 32332 tablet_bootstrap.cc:654] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.705896 32332 tablet_bootstrap.cc:492] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:43.706478 32332 ts_tablet_manager.cc:1367] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
W20240503 14:13:43.706864 28238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 49775.4us
I20240503 14:13:43.708407 32328 tablet_bootstrap.cc:492] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:43.709249 32332 raft_consensus.cc:348] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.710023 32332 raft_consensus.cc:374] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.710107 32332 raft_consensus.cc:729] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.710270 32328 tablet_bootstrap.cc:654] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.710405 32332 consensus_queue.cc:260] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.711356 32334 tablet_bootstrap.cc:654] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:43.711670 28109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 50581.3us
I20240503 14:13:43.713583 32332 ts_tablet_manager.cc:1398] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:43.719636 27899 catalog_manager.cc:5484] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:43.723506 32328 tablet_bootstrap.cc:492] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:43.723717 32328 ts_tablet_manager.cc:1367] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240503 14:13:43.724701 32328 raft_consensus.cc:348] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.724994 32328 raft_consensus.cc:374] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.725180 32328 raft_consensus.cc:729] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.725226 32245 raft_consensus.cc:482] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.725399 32245 raft_consensus.cc:504] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.725466 32328 consensus_queue.cc:260] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.726025 32328 ts_tablet_manager.cc:1398] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:43.726387 32245 leader_election.cc:288] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:43.727406 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:43.727474 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:43.727659 31071 raft_consensus.cc:2454] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:43.728080 30876 leader_election.cc:302] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
W20240503 14:13:43.728277 30875 leader_election.cc:341] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:43.728734 32245 raft_consensus.cc:2781] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.728830 32245 raft_consensus.cc:482] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.728940 32245 raft_consensus.cc:3037] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.731226 32333 tablet_bootstrap.cc:654] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.731583 32245 raft_consensus.cc:504] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.733084 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
W20240503 14:13:43.733552 30875 leader_election.cc:341] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:43.734212 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:43.734411 31071 raft_consensus.cc:3037] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.737540 32333 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:43.737754 32333 ts_tablet_manager.cc:1367] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.003s
I20240503 14:13:43.737787 32334 tablet_bootstrap.cc:492] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:43.738090 31071 raft_consensus.cc:2454] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:43.737929 32334 ts_tablet_manager.cc:1367] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.000s
I20240503 14:13:43.738478 30876 leader_election.cc:302] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters: 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:13:43.738744 32190 raft_consensus.cc:2781] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.738873 32190 raft_consensus.cc:686] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:43.739055 32333 raft_consensus.cc:348] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.739274 32333 raft_consensus.cc:374] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.739138 32190 consensus_queue.cc:237] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.739409 32333 raft_consensus.cc:729] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.739610 32245 leader_election.cc:288] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:43.739657 32333 consensus_queue.cc:260] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.739676 32334 raft_consensus.cc:348] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.739962 32334 raft_consensus.cc:374] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.740088 32334 raft_consensus.cc:729] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.740269 32333 ts_tablet_manager.cc:1398] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:43.740384 32334 consensus_queue.cc:260] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.740602 32333 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:43.742292 32333 tablet_bootstrap.cc:654] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.745833 32334 ts_tablet_manager.cc:1398] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:43.751477 32333 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:43.751657 32333 ts_tablet_manager.cc:1367] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 14:13:43.752501 32333 raft_consensus.cc:348] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.752789 32333 raft_consensus.cc:374] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.752911 32333 raft_consensus.cc:729] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.753217 32333 consensus_queue.cc:260] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.753504 27899 catalog_manager.cc:5484] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:43.754914 32333 ts_tablet_manager.cc:1398] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:43.793627 32280 raft_consensus.cc:482] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.793835 32280 raft_consensus.cc:504] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.794675 32280 leader_election.cc:288] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:43.795121 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a82a00c33e1412f93ab071d5b0d824a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:43.795176 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a82a00c33e1412f93ab071d5b0d824a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:43.795341 30809 raft_consensus.cc:2454] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:43.795385 30939 raft_consensus.cc:2454] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:43.795754 31136 leader_election.cc:302] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:43.796010 32280 raft_consensus.cc:2781] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.796097 32280 raft_consensus.cc:482] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.796164 32280 raft_consensus.cc:3037] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.798015 32280 raft_consensus.cc:504] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.798549 32280 leader_election.cc:288] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:43.799000 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a82a00c33e1412f93ab071d5b0d824a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:43.798992 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a82a00c33e1412f93ab071d5b0d824a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:43.799206 30809 raft_consensus.cc:3037] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.799187 30939 raft_consensus.cc:3037] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.802109 30939 raft_consensus.cc:2454] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:43.802626 31138 leader_election.cc:302] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:43.802958 30809 raft_consensus.cc:2454] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:43.803812 32280 raft_consensus.cc:2781] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.804533 32280 raft_consensus.cc:686] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:43.804783 32280 consensus_queue.cc:237] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:43.808301 27899 catalog_manager.cc:5484] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:43.829334 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:43.834636 31071 raft_consensus.cc:1261] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.834697 30940 raft_consensus.cc:1261] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.835240 32344 consensus_queue.cc:1035] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.835608 32345 consensus_queue.cc:1035] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.840741 27899 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 7a648901743a4d109bfb2309afc4f4ca (table rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31]). Expected version 1 got 0
I20240503 14:13:43.841884 32356 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.842031 32357 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.842518 32357 tablet.cc:1620] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.842515 32356 tablet.cc:1620] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:43.844928 27901 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 7a648901743a4d109bfb2309afc4f4ca (table rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31]). Expected version 1 got 0
I20240503 14:13:43.851074 32358 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.851580 32358 tablet.cc:1620] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.852092 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31] alter complete (version 1)
I20240503 14:13:43.854749 27901 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 7a648901743a4d109bfb2309afc4f4ca (table rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31]). Expected version 1 got 0
I20240503 14:13:43.929724 32341 raft_consensus.cc:482] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.929926 32341 raft_consensus.cc:504] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.930786 32341 leader_election.cc:288] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:43.931425 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611bac13cf32492b87c2f57752c59462" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:43.931488 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611bac13cf32492b87c2f57752c59462" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:43.931638 30809 raft_consensus.cc:2454] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:43.931672 28040 raft_consensus.cc:2454] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:43.932061 28237 leader_election.cc:302] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:43.932299 32341 raft_consensus.cc:2781] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.932401 32341 raft_consensus.cc:482] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.932492 32341 raft_consensus.cc:3037] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.934967 32341 raft_consensus.cc:504] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.935452 32341 leader_election.cc:288] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:43.935844 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611bac13cf32492b87c2f57752c59462" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:43.935880 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611bac13cf32492b87c2f57752c59462" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:43.936013 28040 raft_consensus.cc:3037] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.936033 30809 raft_consensus.cc:3037] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.938468 32280 raft_consensus.cc:482] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.938613 28040 raft_consensus.cc:2454] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:43.938818 32280 raft_consensus.cc:504] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:43.939309 28240 leader_election.cc:302] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:43.939882 32341 raft_consensus.cc:2781] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.940095 32341 raft_consensus.cc:686] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:43.940078 32280 leader_election.cc:288] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:43.940217 30809 raft_consensus.cc:2454] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:43.940557 32341 consensus_queue.cc:237] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:43.941565 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:43.942013 31071 raft_consensus.cc:2379] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 530893a961af4b95b6b687fa5ed22f9d in current term 1: Already voted for candidate 6bbc686598f24f518d7ca6bb2063058d in this term.
I20240503 14:13:43.947010 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:43.947669 31138 leader_election.cc:302] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:43.947558 27899 catalog_manager.cc:5484] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:43.948210 32280 raft_consensus.cc:3037] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.951443 32280 raft_consensus.cc:2726] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:43.957461 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:43.963873 32356 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.964332 32356 tablet.cc:1620] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:43.964551 32357 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.965018 32357 tablet.cc:1620] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.971287 32358 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.971789 32358 tablet.cc:1620] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:43.973631 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=40596cb076554ee090582c8e3d30ae31] alter complete (version 2)
I20240503 14:13:44.076004 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:44.082841 32356 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:44.084290 32358 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:44.084079 32357 tablet.cc:1722] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:44.092664 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=40596cb076554ee090582c8e3d30ae31] alter complete (version 3)
I20240503 14:13:44.151086 32245 consensus_queue.cc:1035] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.155567 32190 consensus_queue.cc:1035] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.195518 27900 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:44.195739 27900 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:44.198751 27900 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 7a648901743a4d109bfb2309afc4f4ca
I20240503 14:13:44.199425 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a648901743a4d109bfb2309afc4f4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:44 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:44.199537 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a648901743a4d109bfb2309afc4f4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:44 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:44.199563 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a648901743a4d109bfb2309afc4f4ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:44 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:44.200246 32364 tablet_replica.cc:330] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:44.200515 32364 raft_consensus.cc:2229] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:44.200799 32364 raft_consensus.cc:2258] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:44.201412 32365 tablet_replica.cc:330] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:44.201659 32365 raft_consensus.cc:2229] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:44.201666 32363 tablet_replica.cc:330] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:44.201792 32364 ts_tablet_manager.cc:1875] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:44.201870 32363 raft_consensus.cc:2229] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:44.202054 32365 raft_consensus.cc:2258] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:44.202085 32363 raft_consensus.cc:2258] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:44.202796 32365 ts_tablet_manager.cc:1875] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:44.202992 32363 ts_tablet_manager.cc:1875] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:44.208523 32364 ts_tablet_manager.cc:1888] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:44.208658 32364 log.cc:1196] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/7a648901743a4d109bfb2309afc4f4ca
I20240503 14:13:44.208997 32363 ts_tablet_manager.cc:1888] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:44.209139 32364 ts_tablet_manager.cc:1909] T 7a648901743a4d109bfb2309afc4f4ca P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:44.209461 32365 ts_tablet_manager.cc:1888] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:44.209574 32365 log.cc:1196] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/7a648901743a4d109bfb2309afc4f4ca
I20240503 14:13:44.209688 32363 log.cc:1196] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/7a648901743a4d109bfb2309afc4f4ca
I20240503 14:13:44.210021 32365 ts_tablet_manager.cc:1909] T 7a648901743a4d109bfb2309afc4f4ca P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:44.210271 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 7a648901743a4d109bfb2309afc4f4ca (table rebalancer_dynamic_table_new_name [id=40596cb076554ee090582c8e3d30ae31]) successfully deleted
I20240503 14:13:44.210336 32363 ts_tablet_manager.cc:1909] T 7a648901743a4d109bfb2309afc4f4ca P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:44.210929 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 7a648901743a4d109bfb2309afc4f4ca (table rebalancer_dynamic_table_new_name [id=40596cb076554ee090582c8e3d30ae31]) successfully deleted
I20240503 14:13:44.211249 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 7a648901743a4d109bfb2309afc4f4ca (table rebalancer_dynamic_table_new_name [id=40596cb076554ee090582c8e3d30ae31]) successfully deleted
I20240503 14:13:44.244988 32280 consensus_queue.cc:1035] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.248965 32280 consensus_queue.cc:1035] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.300978 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:44.309080 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 6aeba244905e4e7dbecf1c1619d2799e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7dc879ed1c744286889d9fba85105ea5]), partition=
I20240503 14:13:44.309541 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aeba244905e4e7dbecf1c1619d2799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.309671 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 6aeba244905e4e7dbecf1c1619d2799e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7dc879ed1c744286889d9fba85105ea5]), partition=
I20240503 14:13:44.310022 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aeba244905e4e7dbecf1c1619d2799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.314937 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 6aeba244905e4e7dbecf1c1619d2799e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7dc879ed1c744286889d9fba85105ea5]), partition=
I20240503 14:13:44.315315 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aeba244905e4e7dbecf1c1619d2799e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.321394 32371 tablet_bootstrap.cc:492] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:44.323119 32370 tablet_bootstrap.cc:492] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:44.323335 32371 tablet_bootstrap.cc:654] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.323698 32369 tablet_bootstrap.cc:492] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:44.324854 32370 tablet_bootstrap.cc:654] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.325309 32369 tablet_bootstrap.cc:654] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.325320 32371 tablet_bootstrap.cc:492] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:44.325464 32371 ts_tablet_manager.cc:1367] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:44.326721 32370 tablet_bootstrap.cc:492] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:44.326869 32370 ts_tablet_manager.cc:1367] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:44.326853 32371 raft_consensus.cc:348] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.327116 32371 raft_consensus.cc:374] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.327216 32371 raft_consensus.cc:729] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.327447 32371 consensus_queue.cc:260] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.327562 32370 raft_consensus.cc:348] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.327739 32370 raft_consensus.cc:374] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.327841 32370 raft_consensus.cc:729] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.328047 32369 tablet_bootstrap.cc:492] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:44.328065 32370 consensus_queue.cc:260] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.328186 32369 ts_tablet_manager.cc:1367] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:13:44.329044 32369 raft_consensus.cc:348] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.329339 32369 raft_consensus.cc:374] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.328572 32370 ts_tablet_manager.cc:1398] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:44.331137 32371 ts_tablet_manager.cc:1398] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:44.334479 32369 raft_consensus.cc:729] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.334805 32369 consensus_queue.cc:260] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.335278 32369 ts_tablet_manager.cc:1398] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:13:44.392941 32230 raft_consensus.cc:482] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.393139 32230 raft_consensus.cc:504] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.394374 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeba244905e4e7dbecf1c1619d2799e" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:44.394380 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeba244905e4e7dbecf1c1619d2799e" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:44.394609 31203 raft_consensus.cc:2454] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:44.394618 31071 raft_consensus.cc:2454] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:44.395049 28369 leader_election.cc:302] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:44.395349 32307 raft_consensus.cc:2781] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.395468 32307 raft_consensus.cc:482] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.395479 32230 leader_election.cc:288] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:44.395603 32307 raft_consensus.cc:3037] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.398275 32307 raft_consensus.cc:504] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.398662 32341 consensus_queue.cc:1035] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.398861 32307 leader_election.cc:288] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:44.399353 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeba244905e4e7dbecf1c1619d2799e" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:44.399494 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aeba244905e4e7dbecf1c1619d2799e" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:44.399555 31203 raft_consensus.cc:3037] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.399670 31071 raft_consensus.cc:3037] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.402272 31203 raft_consensus.cc:2454] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:44.402736 28368 leader_election.cc:302] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:13:44.403012 32307 raft_consensus.cc:2781] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.403295 32307 raft_consensus.cc:686] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:44.403270 31071 raft_consensus.cc:2454] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:44.407496 32307 consensus_queue.cc:237] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.411156 32341 consensus_queue.cc:1035] T 611bac13cf32492b87c2f57752c59462 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.412214 27901 catalog_manager.cc:5484] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:44.454538 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:44.463040 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b0218ec01450f94a394efb7d82149 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.463553 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b0218ec01450f94a394efb7d82149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.463729 30789 tablet_service.cc:1467] Processing CreateTablet for tablet f11b07607b0e430db646172f863cdc1f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.463743 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b0218ec01450f94a394efb7d82149 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.464154 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11b07607b0e430db646172f863cdc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.464183 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b0218ec01450f94a394efb7d82149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.467041 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6b0218ec01450f94a394efb7d82149 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.467450 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6b0218ec01450f94a394efb7d82149. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.469058 32382 tablet_bootstrap.cc:492] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:44.469892 28154 tablet_service.cc:1467] Processing CreateTablet for tablet f11b07607b0e430db646172f863cdc1f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.470343 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11b07607b0e430db646172f863cdc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.471007 32382 tablet_bootstrap.cc:654] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.475426 31051 tablet_service.cc:1467] Processing CreateTablet for tablet f11b07607b0e430db646172f863cdc1f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.476001 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11b07607b0e430db646172f863cdc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.480122 32371 tablet_bootstrap.cc:492] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:44.481961 32371 tablet_bootstrap.cc:654] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.485177 32385 tablet_bootstrap.cc:492] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:44.485899 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 96773dd7242d4223bdc663ee53bfdd3f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.486328 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96773dd7242d4223bdc663ee53bfdd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.487010 32385 tablet_bootstrap.cc:654] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.488795 32383 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:44.490422 32383 tablet_bootstrap.cc:654] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.494608 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 96773dd7242d4223bdc663ee53bfdd3f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.495168 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96773dd7242d4223bdc663ee53bfdd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.496691 32385 tablet_bootstrap.cc:492] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:44.496812 32385 ts_tablet_manager.cc:1367] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240503 14:13:44.497661 32385 raft_consensus.cc:348] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.497875 32385 raft_consensus.cc:374] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.497977 32385 raft_consensus.cc:729] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.498844 32385 consensus_queue.cc:260] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.499843 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 96773dd7242d4223bdc663ee53bfdd3f (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:44.500285 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96773dd7242d4223bdc663ee53bfdd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.502724 32385 ts_tablet_manager.cc:1398] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:13:44.507897 32369 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:44.509675 32369 tablet_bootstrap.cc:654] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.512579 32370 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:44.514703 32370 tablet_bootstrap.cc:654] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.514804 32383 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:44.514966 32383 ts_tablet_manager.cc:1367] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240503 14:13:44.516109 32383 raft_consensus.cc:348] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.516376 32383 raft_consensus.cc:374] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.516494 32383 raft_consensus.cc:729] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.516772 32383 consensus_queue.cc:260] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.517422 32383 ts_tablet_manager.cc:1398] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:44.519524 32383 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:44.521238 32383 tablet_bootstrap.cc:654] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.521431 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:44.527647 32371 tablet_bootstrap.cc:492] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:44.527796 32371 ts_tablet_manager.cc:1367] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.006s
I20240503 14:13:44.528481 32371 raft_consensus.cc:348] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.529672 32371 raft_consensus.cc:374] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.529896 32371 raft_consensus.cc:729] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.530531 32371 consensus_queue.cc:260] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.531322 32371 ts_tablet_manager.cc:1398] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:44.534313 32370 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:44.534564 32370 ts_tablet_manager.cc:1367] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20240503 14:13:44.535485 32370 raft_consensus.cc:348] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.536031 32370 raft_consensus.cc:374] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.536170 32370 raft_consensus.cc:729] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.536502 32370 consensus_queue.cc:260] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.544538 32369 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:44.545346 31203 raft_consensus.cc:1261] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.545936 32230 consensus_queue.cc:1035] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.548897 32369 ts_tablet_manager.cc:1367] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.014s
I20240503 14:13:44.553263 31072 raft_consensus.cc:1261] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.551751 32369 raft_consensus.cc:348] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.554069 32369 raft_consensus.cc:374] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.554500 32369 raft_consensus.cc:729] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.554921 32230 consensus_queue.cc:1035] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.555404 32369 consensus_queue.cc:260] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.556731 32369 ts_tablet_manager.cc:1398] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:13:44.567456 32382 tablet_bootstrap.cc:492] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:44.567703 32382 ts_tablet_manager.cc:1367] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.099s user 0.020s sys 0.020s
I20240503 14:13:44.568770 32382 raft_consensus.cc:348] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.569109 32382 raft_consensus.cc:374] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.569257 32382 raft_consensus.cc:729] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.569553 32382 consensus_queue.cc:260] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.570431 32382 ts_tablet_manager.cc:1398] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:44.591617 32230 raft_consensus.cc:482] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.591915 32230 raft_consensus.cc:504] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.592779 32230 leader_election.cc:288] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:44.593780 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96773dd7242d4223bdc663ee53bfdd3f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
W20240503 14:13:44.595307 28369 leader_election.cc:341] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.595386 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96773dd7242d4223bdc663ee53bfdd3f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
W20240503 14:13:44.596181 28367 leader_election.cc:341] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.596354 28367 leader_election.cc:302] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:44.596743 32230 raft_consensus.cc:2726] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:44.600454 32370 ts_tablet_manager.cc:1398] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.066s user 0.002s sys 0.000s
I20240503 14:13:44.600955 32370 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:44.649742 32383 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:44.649956 32383 ts_tablet_manager.cc:1367] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.130s user 0.030s sys 0.070s
I20240503 14:13:44.650658 32383 raft_consensus.cc:348] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.650856 32383 raft_consensus.cc:374] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.650969 32383 raft_consensus.cc:729] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.651199 32383 consensus_queue.cc:260] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.658474 27901 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 6aeba244905e4e7dbecf1c1619d2799e (table rebalancer_dynamic_table [id=7dc879ed1c744286889d9fba85105ea5]). Expected version 1 got 0
I20240503 14:13:44.667990 32401 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:44.668675 32401 tablet.cc:1620] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:44.669514 32402 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:44.670135 32402 tablet.cc:1620] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:44.670410 32370 tablet_bootstrap.cc:654] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.672289 32390 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:44.673900 32390 tablet_bootstrap.cc:654] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.679738 32405 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:44.680370 32405 tablet.cc:1620] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:44.681964 32370 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:44.682024 27900 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 6aeba244905e4e7dbecf1c1619d2799e (table rebalancer_dynamic_table [id=7dc879ed1c744286889d9fba85105ea5]). Expected version 1 got 0
I20240503 14:13:44.682112 32370 ts_tablet_manager.cc:1367] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.000s
I20240503 14:13:44.682762 32370 raft_consensus.cc:348] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.683077 32370 raft_consensus.cc:374] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.683178 32370 raft_consensus.cc:729] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.683404 32370 consensus_queue.cc:260] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.683921 32370 ts_tablet_manager.cc:1398] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:44.690459 32390 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:44.690702 32390 ts_tablet_manager.cc:1367] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20240503 14:13:44.691612 32390 raft_consensus.cc:348] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.691948 32390 raft_consensus.cc:374] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.692126 32390 raft_consensus.cc:729] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.692420 32390 consensus_queue.cc:260] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.693136 32390 ts_tablet_manager.cc:1398] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:44.694684 32383 ts_tablet_manager.cc:1398] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.045s user 0.021s sys 0.017s
I20240503 14:13:44.700213 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=7dc879ed1c744286889d9fba85105ea5] alter complete (version 1)
I20240503 14:13:44.751312 32376 raft_consensus.cc:482] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.751477 32376 raft_consensus.cc:504] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.752061 32376 leader_election.cc:288] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:44.752508 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b0218ec01450f94a394efb7d82149" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:44.752558 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b0218ec01450f94a394efb7d82149" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:44.752728 31201 raft_consensus.cc:2454] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:44.752728 30939 raft_consensus.cc:2454] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:44.753067 28239 leader_election.cc:302] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:44.753322 32376 raft_consensus.cc:2781] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.753437 32376 raft_consensus.cc:482] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.753521 32376 raft_consensus.cc:3037] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.757228 32376 raft_consensus.cc:504] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.757761 32376 leader_election.cc:288] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:44.758126 31201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b0218ec01450f94a394efb7d82149" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:44.758332 31201 raft_consensus.cc:3037] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.758388 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6b0218ec01450f94a394efb7d82149" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:44.758605 30939 raft_consensus.cc:3037] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.760478 31201 raft_consensus.cc:2454] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:44.760787 28238 leader_election.cc:302] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:44.760972 30939 raft_consensus.cc:2454] T 0a6b0218ec01450f94a394efb7d82149 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:44.761026 32376 raft_consensus.cc:2781] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.761164 32376 raft_consensus.cc:686] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:44.761449 32376 consensus_queue.cc:237] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:44.765298 27899 catalog_manager.cc:5484] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:44.792666 32345 raft_consensus.cc:482] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.792886 32345 raft_consensus.cc:504] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.793576 32345 leader_election.cc:288] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:44.794188 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11b07607b0e430db646172f863cdc1f" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:44.794337 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11b07607b0e430db646172f863cdc1f" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:44.794517 31072 raft_consensus.cc:2454] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:44.794581 30810 raft_consensus.cc:2454] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:44.794999 28109 leader_election.cc:302] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:44.795277 32345 raft_consensus.cc:2781] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.795362 32345 raft_consensus.cc:482] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.795469 32345 raft_consensus.cc:3037] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.797968 32345 raft_consensus.cc:504] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.798502 32345 leader_election.cc:288] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:44.798825 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11b07607b0e430db646172f863cdc1f" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:44.798863 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11b07607b0e430db646172f863cdc1f" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:44.798976 30810 raft_consensus.cc:3037] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.799033 31072 raft_consensus.cc:3037] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.801097 31072 raft_consensus.cc:2454] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:44.801178 30810 raft_consensus.cc:2454] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:44.801530 28107 leader_election.cc:302] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:44.801903 32345 raft_consensus.cc:2781] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.802065 32345 raft_consensus.cc:686] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:44.802415 32345 consensus_queue.cc:237] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:44.806080 27899 catalog_manager.cc:5484] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:44.814101 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:44.820489 32401 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:44.821058 32401 tablet.cc:1620] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:44.821094 32405 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:44.821811 32405 tablet.cc:1620] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:44.824687 32402 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:44.825162 32402 tablet.cc:1620] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:44.830497 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=7dc879ed1c744286889d9fba85105ea5] alter complete (version 2)
I20240503 14:13:44.859982 32400 raft_consensus.cc:482] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.860172 32400 raft_consensus.cc:504] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.861162 32400 leader_election.cc:288] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:44.862399 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96773dd7242d4223bdc663ee53bfdd3f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:44.862805 31072 raft_consensus.cc:2454] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:44.863395 30745 leader_election.cc:302] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:44.863770 32400 raft_consensus.cc:2781] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.863884 32400 raft_consensus.cc:482] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.863996 32400 raft_consensus.cc:3037] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.867761 32400 raft_consensus.cc:504] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.869103 32400 leader_election.cc:288] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:44.869277 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96773dd7242d4223bdc663ee53bfdd3f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:44.869501 31071 raft_consensus.cc:3037] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.873051 31071 raft_consensus.cc:2454] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:44.873478 30745 leader_election.cc:302] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:44.873852 32400 raft_consensus.cc:2781] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.874024 32400 raft_consensus.cc:686] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:44.874786 32400 consensus_queue.cc:237] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:44.879947 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96773dd7242d4223bdc663ee53bfdd3f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:44.880182 28432 raft_consensus.cc:2454] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:44.881458 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96773dd7242d4223bdc663ee53bfdd3f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:44.881641 28432 raft_consensus.cc:3037] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.885648 28432 raft_consensus.cc:2454] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:44.891106 27900 catalog_manager.cc:5484] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:44.935081 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:44.942647 32401 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:44.943735 32405 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:44.944331 32402 tablet.cc:1722] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:44.955272 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=7dc879ed1c744286889d9fba85105ea5] alter complete (version 3)
I20240503 14:13:45.066923 27900 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:45.067235 27900 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:45.071094 27900 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 6aeba244905e4e7dbecf1c1619d2799e
I20240503 14:13:45.071995 31181 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aeba244905e4e7dbecf1c1619d2799e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:45.072192 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aeba244905e4e7dbecf1c1619d2799e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:45.072228 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aeba244905e4e7dbecf1c1619d2799e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:45.085590 32414 tablet_replica.cc:330] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:45.085983 32412 tablet_replica.cc:330] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:45.086087 32414 raft_consensus.cc:2229] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:45.086432 32414 raft_consensus.cc:2258] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.086448 32412 raft_consensus.cc:2229] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:45.087426 32412 raft_consensus.cc:2258] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.087548 32414 ts_tablet_manager.cc:1875] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.089852 32412 ts_tablet_manager.cc:1875] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.092135 32413 tablet_replica.cc:330] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:45.092435 32413 raft_consensus.cc:2229] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:45.092866 32413 raft_consensus.cc:2258] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.093930 32413 ts_tablet_manager.cc:1875] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.096915 32412 ts_tablet_manager.cc:1888] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.097044 32412 log.cc:1196] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/6aeba244905e4e7dbecf1c1619d2799e
I20240503 14:13:45.097496 32412 ts_tablet_manager.cc:1909] T 6aeba244905e4e7dbecf1c1619d2799e P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:45.098224 32414 ts_tablet_manager.cc:1888] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.098398 32414 log.cc:1196] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/6aeba244905e4e7dbecf1c1619d2799e
I20240503 14:13:45.098526 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 6aeba244905e4e7dbecf1c1619d2799e (table rebalancer_dynamic_table_new_name [id=7dc879ed1c744286889d9fba85105ea5]) successfully deleted
I20240503 14:13:45.098996 32414 ts_tablet_manager.cc:1909] T 6aeba244905e4e7dbecf1c1619d2799e P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:45.100224 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 6aeba244905e4e7dbecf1c1619d2799e (table rebalancer_dynamic_table_new_name [id=7dc879ed1c744286889d9fba85105ea5]) successfully deleted
I20240503 14:13:45.102134 32413 ts_tablet_manager.cc:1888] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.102265 32413 log.cc:1196] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/6aeba244905e4e7dbecf1c1619d2799e
I20240503 14:13:45.102604 32413 ts_tablet_manager.cc:1909] T 6aeba244905e4e7dbecf1c1619d2799e P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:45.103474 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 6aeba244905e4e7dbecf1c1619d2799e (table rebalancer_dynamic_table_new_name [id=7dc879ed1c744286889d9fba85105ea5]) successfully deleted
I20240503 14:13:45.180459 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:45.189239 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 6886a0458c884b63bc993636f27a2c0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54]), partition=
I20240503 14:13:45.189579 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6886a0458c884b63bc993636f27a2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.193964 32383 tablet_bootstrap.cc:492] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:45.194808 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 6886a0458c884b63bc993636f27a2c0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54]), partition=
I20240503 14:13:45.195178 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6886a0458c884b63bc993636f27a2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.195530 32383 tablet_bootstrap.cc:654] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.199295 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 6886a0458c884b63bc993636f27a2c0a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54]), partition=
I20240503 14:13:45.199676 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6886a0458c884b63bc993636f27a2c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.216820 32383 tablet_bootstrap.cc:492] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:45.217057 32383 ts_tablet_manager.cc:1367] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.003s
I20240503 14:13:45.217916 32383 raft_consensus.cc:348] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.219152 32383 raft_consensus.cc:374] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.219339 32383 raft_consensus.cc:729] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.219648 32383 consensus_queue.cc:260] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.220585 32383 ts_tablet_manager.cc:1398] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:45.230782 32416 tablet_bootstrap.cc:492] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:45.233297 32416 tablet_bootstrap.cc:654] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.234809 32417 tablet_bootstrap.cc:492] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:45.236119 32416 tablet_bootstrap.cc:492] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:45.237588 32416 ts_tablet_manager.cc:1367] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:13:45.238175 32417 tablet_bootstrap.cc:654] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.238749 32416 raft_consensus.cc:348] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.238925 32416 raft_consensus.cc:374] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.239023 32416 raft_consensus.cc:729] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.239275 32416 consensus_queue.cc:260] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.240334 32417 tablet_bootstrap.cc:492] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:45.240483 32417 ts_tablet_manager.cc:1367] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:45.241484 32417 raft_consensus.cc:348] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.241710 32417 raft_consensus.cc:374] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.241798 32417 raft_consensus.cc:729] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.242067 32417 consensus_queue.cc:260] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.242537 32416 ts_tablet_manager.cc:1398] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:45.244091 32417 ts_tablet_manager.cc:1398] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:45.255458 32325 consensus_queue.cc:1035] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.271905 32400 consensus_queue.cc:1035] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.274956 32345 consensus_queue.cc:1035] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.289789 32425 consensus_queue.cc:1035] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.308880 32426 raft_consensus.cc:482] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.309067 32426 raft_consensus.cc:504] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.310184 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6886a0458c884b63bc993636f27a2c0a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:45.310439 31071 raft_consensus.cc:2454] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:45.310581 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6886a0458c884b63bc993636f27a2c0a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:45.310833 30810 raft_consensus.cc:2454] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:45.310868 28109 leader_election.cc:302] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:45.311379 32345 raft_consensus.cc:2781] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:45.311527 32345 raft_consensus.cc:482] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:45.311635 32345 raft_consensus.cc:3037] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.315724 32376 consensus_queue.cc:1035] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.319739 32345 raft_consensus.cc:504] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.320816 32426 leader_election.cc:288] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:45.321080 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6886a0458c884b63bc993636f27a2c0a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:45.321303 31071 raft_consensus.cc:3037] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.321447 32345 leader_election.cc:288] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:45.323740 31071 raft_consensus.cc:2454] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:45.324167 28109 leader_election.cc:302] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:45.324520 32345 raft_consensus.cc:2781] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:45.324661 32345 raft_consensus.cc:686] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:45.324617 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6886a0458c884b63bc993636f27a2c0a" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:45.324867 30810 raft_consensus.cc:3037] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.324939 32345 consensus_queue.cc:237] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:45.328895 30810 raft_consensus.cc:2454] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:45.331802 32125 consensus_queue.cc:1035] T 0a6b0218ec01450f94a394efb7d82149 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.364771 27900 catalog_manager.cc:5484] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:45.422384 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:45.431133 31181 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb8c05985f34deebc145c3c51dc413d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.431578 31181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb8c05985f34deebc145c3c51dc413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.431492 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb8c05985f34deebc145c3c51dc413d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.431882 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb8c05985f34deebc145c3c51dc413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.433408 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 027c88573444465a9ebcaa9589728e74 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.433815 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027c88573444465a9ebcaa9589728e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.434907 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 027c88573444465a9ebcaa9589728e74 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.435276 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027c88573444465a9ebcaa9589728e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.435390 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb8c05985f34deebc145c3c51dc413d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.435848 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb8c05985f34deebc145c3c51dc413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.438664 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 027c88573444465a9ebcaa9589728e74 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.439093 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027c88573444465a9ebcaa9589728e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.442050 31182 tablet_service.cc:1467] Processing CreateTablet for tablet e389fcad67cc4add9db2f34eb57745c8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.442287 32383 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:45.442449 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e389fcad67cc4add9db2f34eb57745c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.443866 32383 tablet_bootstrap.cc:654] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.444595 32416 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:45.446096 32416 tablet_bootstrap.cc:654] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.447589 28022 tablet_service.cc:1467] Processing CreateTablet for tablet e389fcad67cc4add9db2f34eb57745c8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.448112 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e389fcad67cc4add9db2f34eb57745c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.454813 32436 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:45.456411 32436 tablet_bootstrap.cc:654] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.459265 32383 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:45.459424 32383 ts_tablet_manager.cc:1367] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20240503 14:13:45.459717 32416 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:45.459885 32416 ts_tablet_manager.cc:1367] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240503 14:13:45.460062 32383 raft_consensus.cc:348] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.460247 32383 raft_consensus.cc:374] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.460315 32383 raft_consensus.cc:729] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.460705 32383 consensus_queue.cc:260] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.461256 32383 ts_tablet_manager.cc:1398] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:45.462754 32416 raft_consensus.cc:348] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.462960 32416 raft_consensus.cc:374] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.463028 32416 raft_consensus.cc:729] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.463235 32416 consensus_queue.cc:260] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.463809 32416 ts_tablet_manager.cc:1398] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:45.464298 32416 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:45.465852 32416 tablet_bootstrap.cc:654] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.473752 32400 raft_consensus.cc:482] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.473912 32400 raft_consensus.cc:504] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.474066 32416 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:45.474224 32416 ts_tablet_manager.cc:1367] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 14:13:45.474727 32400 leader_election.cc:288] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:45.475064 32416 raft_consensus.cc:348] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.475262 32416 raft_consensus.cc:374] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.475351 32416 raft_consensus.cc:729] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.475564 32416 consensus_queue.cc:260] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.476148 32416 ts_tablet_manager.cc:1398] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:45.477550 31050 tablet_service.cc:1467] Processing CreateTablet for tablet e389fcad67cc4add9db2f34eb57745c8 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.477994 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e389fcad67cc4add9db2f34eb57745c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.487306 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240503 14:13:45.491438 30744 leader_election.cc:341] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Not found: Tablet not found: 027c88573444465a9ebcaa9589728e74
I20240503 14:13:45.496997 32436 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:45.497186 32436 ts_tablet_manager.cc:1367] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240503 14:13:45.498139 32436 raft_consensus.cc:348] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.498404 32436 raft_consensus.cc:374] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.498505 32436 raft_consensus.cc:729] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.498757 32436 consensus_queue.cc:260] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.499437 32436 ts_tablet_manager.cc:1398] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:45.502414 31071 raft_consensus.cc:1261] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.502439 30809 raft_consensus.cc:1261] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.503104 32345 consensus_queue.cc:1035] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.505653 32436 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:45.506112 32440 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:45.506179 32345 consensus_queue.cc:1035] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.507896 32436 tablet_bootstrap.cc:654] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.508328 32440 tablet_bootstrap.cc:654] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.509681 32443 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:45.511557 32416 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:45.513093 32443 tablet_bootstrap.cc:654] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.513098 32416 tablet_bootstrap.cc:654] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.517168 27900 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 6886a0458c884b63bc993636f27a2c0a (table rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54]). Expected version 1 got 0
I20240503 14:13:45.517107 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027c88573444465a9ebcaa9589728e74" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:45.517411 31070 raft_consensus.cc:2454] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:45.531600 32449 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:45.532229 32450 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:45.532826 27894 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 6886a0458c884b63bc993636f27a2c0a (table rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54]). Expected version 1 got 0
I20240503 14:13:45.534310 32436 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:45.534475 32436 ts_tablet_manager.cc:1367] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
W20240503 14:13:45.534794 30877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 71107.7us
I20240503 14:13:45.536285 32436 raft_consensus.cc:348] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.536937 30745 leader_election.cc:302] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters: 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:13:45.537156 27900 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 6886a0458c884b63bc993636f27a2c0a (table rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54]). Expected version 1 got 0
I20240503 14:13:45.537483 32325 raft_consensus.cc:2781] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:45.537634 32325 raft_consensus.cc:482] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:45.537796 32325 raft_consensus.cc:3037] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.536624 32351 raft_consensus.cc:482] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.538607 32351 raft_consensus.cc:504] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.539245 32351 leader_election.cc:288] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:45.539573 32436 raft_consensus.cc:374] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.539659 32436 raft_consensus.cc:729] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.539891 32436 consensus_queue.cc:260] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.540406 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e389fcad67cc4add9db2f34eb57745c8" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
W20240503 14:13:45.541714 31139 leader_election.cc:341] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:45.541982 32450 tablet.cc:1620] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:45.542500 32416 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:45.542806 32416 ts_tablet_manager.cc:1367] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.007s
I20240503 14:13:45.542866 32449 tablet.cc:1620] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:45.543613 32416 raft_consensus.cc:348] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.543874 32416 raft_consensus.cc:374] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.543859 27900 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 6886a0458c884b63bc993636f27a2c0a (table rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54]). Expected version 1 got 0
I20240503 14:13:45.544018 32416 raft_consensus.cc:729] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.544289 32416 consensus_queue.cc:260] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.544895 32416 ts_tablet_manager.cc:1398] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:45.544832 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e389fcad67cc4add9db2f34eb57745c8" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:45.546254 31072 raft_consensus.cc:2454] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:45.546677 31138 leader_election.cc:302] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:13:45.546921 32351 raft_consensus.cc:2781] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:45.547029 32351 raft_consensus.cc:482] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:45.547109 32351 raft_consensus.cc:3037] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.549607 32351 raft_consensus.cc:504] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.550216 32443 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:45.550405 32443 ts_tablet_manager.cc:1367] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.001s
I20240503 14:13:45.550702 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e389fcad67cc4add9db2f34eb57745c8" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
W20240503 14:13:45.551542 31139 leader_election.cc:341] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:45.551859 32351 leader_election.cc:288] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:45.552809 32325 raft_consensus.cc:504] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.552919 32451 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:45.553505 32451 tablet.cc:1620] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:45.553542 32325 leader_election.cc:288] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:45.554039 32436 ts_tablet_manager.cc:1398] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 14:13:45.554718 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027c88573444465a9ebcaa9589728e74" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:45.554951 31072 raft_consensus.cc:3037] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.557319 32443 raft_consensus.cc:348] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.557745 32443 raft_consensus.cc:374] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.557878 32443 raft_consensus.cc:729] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.558346 32443 consensus_queue.cc:260] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.559080 32443 ts_tablet_manager.cc:1398] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:13:45.559868 31072 raft_consensus.cc:2454] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:45.560599 30745 leader_election.cc:302] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:45.560881 32325 raft_consensus.cc:2781] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:45.561020 32325 raft_consensus.cc:686] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:45.561187 32436 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:45.561254 32325 consensus_queue.cc:237] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.562736 32436 tablet_bootstrap.cc:654] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.574255 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54] alter complete (version 1)
I20240503 14:13:45.575642 32440 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:45.575838 32440 ts_tablet_manager.cc:1367] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.000s
I20240503 14:13:45.578524 32440 raft_consensus.cc:348] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.579759 32440 raft_consensus.cc:374] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.579912 32440 raft_consensus.cc:729] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.580201 32440 consensus_queue.cc:260] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.584511 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027c88573444465a9ebcaa9589728e74" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:45.585930 32440 ts_tablet_manager.cc:1398] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
W20240503 14:13:45.586845 30744 leader_election.cc:341] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:45.592223 32436 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:45.592401 32436 ts_tablet_manager.cc:1367] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240503 14:13:45.593312 32436 raft_consensus.cc:348] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.593600 32436 raft_consensus.cc:374] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.593745 32436 raft_consensus.cc:729] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.594029 32436 consensus_queue.cc:260] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.595182 32436 ts_tablet_manager.cc:1398] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:45.600476 27900 catalog_manager.cc:5484] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:45.602512 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e389fcad67cc4add9db2f34eb57745c8" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:45.602741 31072 raft_consensus.cc:3037] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.609207 31072 raft_consensus.cc:2454] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:45.609736 31138 leader_election.cc:302] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:13:45.610484 32453 raft_consensus.cc:2781] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:45.610839 32453 raft_consensus.cc:686] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:45.611143 32453 consensus_queue.cc:237] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.615134 27899 catalog_manager.cc:5484] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:45.621205 32453 raft_consensus.cc:482] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.621378 32453 raft_consensus.cc:504] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:45.621959 32453 leader_election.cc:288] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:45.622337 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027c88573444465a9ebcaa9589728e74" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:45.622575 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "027c88573444465a9ebcaa9589728e74" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:45.622820 31072 raft_consensus.cc:2379] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 530893a961af4b95b6b687fa5ed22f9d in current term 1: Already voted for candidate d0dbfb0e43f948c3922926aacc365fc1 in this term.
I20240503 14:13:45.623247 31136 leader_election.cc:302] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:45.623502 32453 raft_consensus.cc:3037] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.625784 32453 raft_consensus.cc:2726] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:45.699627 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:45.708950 32449 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:45.709491 32449 tablet.cc:1620] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:45.710392 32451 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:45.710880 32451 tablet.cc:1620] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:45.712576 32450 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:45.713593 32450 tablet.cc:1620] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:45.722957 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=9840b61085204e32bf4e75cc46ef3e54] alter complete (version 2)
I20240503 14:13:45.723685 32453 raft_consensus.cc:482] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.723917 32453 raft_consensus.cc:504] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.725135 32453 leader_election.cc:288] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:45.725159 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb8c05985f34deebc145c3c51dc413d" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:45.725426 31072 raft_consensus.cc:2454] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:45.725657 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb8c05985f34deebc145c3c51dc413d" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:45.725803 31138 leader_election.cc:302] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:45.725950 28432 raft_consensus.cc:2454] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:45.726097 32453 raft_consensus.cc:2781] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:45.726281 32453 raft_consensus.cc:482] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:45.726646 32453 raft_consensus.cc:3037] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.729254 32453 raft_consensus.cc:504] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.729823 32453 leader_election.cc:288] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:45.730091 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb8c05985f34deebc145c3c51dc413d" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:45.730283 31072 raft_consensus.cc:3037] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.730283 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb8c05985f34deebc145c3c51dc413d" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:45.730527 28432 raft_consensus.cc:3037] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.733059 28432 raft_consensus.cc:2454] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:45.733355 31072 raft_consensus.cc:2454] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:45.733404 31139 leader_election.cc:302] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:13:45.733673 32453 raft_consensus.cc:2781] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:45.733806 32453 raft_consensus.cc:686] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:45.734112 32453 consensus_queue.cc:237] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:45.737119 27900 catalog_manager.cc:5484] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:45.824978 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:45.831955 32449 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:45.832489 32451 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:45.834161 32450 tablet.cc:1722] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:45.841234 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=9840b61085204e32bf4e75cc46ef3e54] alter complete (version 3)
I20240503 14:13:45.852288 32399 raft_consensus.cc:482] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.852447 32399 raft_consensus.cc:504] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:45.853212 32399 leader_election.cc:288] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:45.853556 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e389fcad67cc4add9db2f34eb57745c8" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:45.853688 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e389fcad67cc4add9db2f34eb57745c8" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:45.853801 31072 raft_consensus.cc:2379] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 793a0d5dd1634c5faaf0e1e4851e8294 in current term 1: Already voted for candidate 530893a961af4b95b6b687fa5ed22f9d in this term.
I20240503 14:13:45.854291 27977 leader_election.cc:302] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:45.854578 32399 raft_consensus.cc:3037] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.856423 32399 raft_consensus.cc:2726] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:45.946010 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:45.946336 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:45.949440 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 6886a0458c884b63bc993636f27a2c0a
I20240503 14:13:45.950218 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 6886a0458c884b63bc993636f27a2c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:45.950400 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 6886a0458c884b63bc993636f27a2c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:45.950856 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 6886a0458c884b63bc993636f27a2c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:45.952658 32461 tablet_replica.cc:330] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:45.953527 32461 raft_consensus.cc:2229] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:45.953915 32461 raft_consensus.cc:2258] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.955050 32400 consensus_queue.cc:1035] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.959532 32461 ts_tablet_manager.cc:1875] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.960273 32400 consensus_queue.cc:1035] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.962117 32460 tablet_replica.cc:330] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:45.962356 32460 raft_consensus.cc:2229] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:45.962397 32462 tablet_replica.cc:330] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:45.962584 32460 raft_consensus.cc:2258] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.962631 32462 raft_consensus.cc:2229] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:45.963047 32462 raft_consensus.cc:2258] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.963744 32460 ts_tablet_manager.cc:1875] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.964074 32462 ts_tablet_manager.cc:1875] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.970412 32462 ts_tablet_manager.cc:1888] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.970634 32462 log.cc:1196] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/6886a0458c884b63bc993636f27a2c0a
I20240503 14:13:45.971288 32462 ts_tablet_manager.cc:1909] T 6886a0458c884b63bc993636f27a2c0a P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:45.972493 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 6886a0458c884b63bc993636f27a2c0a (table rebalancer_dynamic_table_new_name [id=9840b61085204e32bf4e75cc46ef3e54]) successfully deleted
I20240503 14:13:45.976164 32461 ts_tablet_manager.cc:1888] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.976301 32461 log.cc:1196] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/6886a0458c884b63bc993636f27a2c0a
I20240503 14:13:45.978034 32460 ts_tablet_manager.cc:1888] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.978145 32460 log.cc:1196] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/6886a0458c884b63bc993636f27a2c0a
I20240503 14:13:45.978556 32460 ts_tablet_manager.cc:1909] T 6886a0458c884b63bc993636f27a2c0a P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:45.978668 32461 ts_tablet_manager.cc:1909] T 6886a0458c884b63bc993636f27a2c0a P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:45.979468 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 6886a0458c884b63bc993636f27a2c0a (table rebalancer_dynamic_table_new_name [id=9840b61085204e32bf4e75cc46ef3e54]) successfully deleted
I20240503 14:13:45.979673 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 6886a0458c884b63bc993636f27a2c0a (table rebalancer_dynamic_table_new_name [id=9840b61085204e32bf4e75cc46ef3e54]) successfully deleted
I20240503 14:13:46.052551 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:46.060935 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 764c00a614f94fd48b90ada3b3ea7df1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280]), partition=
I20240503 14:13:46.061317 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 764c00a614f94fd48b90ada3b3ea7df1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280]), partition=
I20240503 14:13:46.061373 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764c00a614f94fd48b90ada3b3ea7df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.061650 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764c00a614f94fd48b90ada3b3ea7df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.063796 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 764c00a614f94fd48b90ada3b3ea7df1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280]), partition=
I20240503 14:13:46.064139 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764c00a614f94fd48b90ada3b3ea7df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.067574 32436 tablet_bootstrap.cc:492] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:46.069877 32436 tablet_bootstrap.cc:654] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.080214 32466 tablet_bootstrap.cc:492] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:46.080924 32468 tablet_bootstrap.cc:492] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:46.081074 32436 tablet_bootstrap.cc:492] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:46.081342 32436 ts_tablet_manager.cc:1367] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.012s
I20240503 14:13:46.082082 32466 tablet_bootstrap.cc:654] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.082865 32468 tablet_bootstrap.cc:654] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.082917 32436 raft_consensus.cc:348] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.083190 32436 raft_consensus.cc:374] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.083278 32436 raft_consensus.cc:729] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.083634 32436 consensus_queue.cc:260] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.084511 32436 ts_tablet_manager.cc:1398] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:46.091090 32466 tablet_bootstrap.cc:492] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:46.091413 32466 ts_tablet_manager.cc:1367] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 14:13:46.092533 32466 raft_consensus.cc:348] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.092836 32466 raft_consensus.cc:374] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.092952 32466 raft_consensus.cc:729] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.093277 32466 consensus_queue.cc:260] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.096030 32468 tablet_bootstrap.cc:492] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:46.096312 32468 ts_tablet_manager.cc:1367] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20240503 14:13:46.096366 32466 ts_tablet_manager.cc:1398] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:46.097113 32468 raft_consensus.cc:348] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.097311 32468 raft_consensus.cc:374] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.097431 32468 raft_consensus.cc:729] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.097644 32468 consensus_queue.cc:260] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.098191 32468 ts_tablet_manager.cc:1398] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:46.135813 32453 consensus_queue.cc:1035] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.140291 32453 consensus_queue.cc:1035] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.152139 32454 raft_consensus.cc:482] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:46.152338 32454 raft_consensus.cc:504] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.152988 32454 leader_election.cc:288] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:46.153414 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764c00a614f94fd48b90ada3b3ea7df1" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:46.153434 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764c00a614f94fd48b90ada3b3ea7df1" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:46.153636 31070 raft_consensus.cc:2454] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:46.153654 30939 raft_consensus.cc:2454] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:46.154070 31138 leader_election.cc:302] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:46.154829 32453 raft_consensus.cc:2781] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:46.154970 32453 raft_consensus.cc:482] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:46.155071 32453 raft_consensus.cc:3037] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.157619 32453 raft_consensus.cc:504] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.158294 32454 consensus_queue.cc:1035] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.158417 32453 leader_election.cc:288] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:46.158831 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764c00a614f94fd48b90ada3b3ea7df1" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:46.159018 31070 raft_consensus.cc:3037] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.159173 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764c00a614f94fd48b90ada3b3ea7df1" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:46.159385 30939 raft_consensus.cc:3037] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.162649 30939 raft_consensus.cc:2454] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:46.163141 31138 leader_election.cc:302] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:46.163502 32454 raft_consensus.cc:2781] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:46.163623 32454 raft_consensus.cc:686] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:46.163877 32454 consensus_queue.cc:237] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.164073 32453 consensus_queue.cc:1035] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.164187 31070 raft_consensus.cc:2454] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:46.168617 27894 catalog_manager.cc:5484] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:46.277657 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:46.285755 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.286352 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.286505 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 8377b1aa704c43b78f310e7440f6488e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.286924 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8377b1aa704c43b78f310e7440f6488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.287353 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.287724 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.290019 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.290630 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.291226 31183 tablet_service.cc:1467] Processing CreateTablet for tablet f3dded25198e4d7d965fd9085e4ccd6d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.291622 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dded25198e4d7d965fd9085e4ccd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.292721 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 8377b1aa704c43b78f310e7440f6488e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.293143 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8377b1aa704c43b78f310e7440f6488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.298792 30789 tablet_service.cc:1467] Processing CreateTablet for tablet f3dded25198e4d7d965fd9085e4ccd6d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.299168 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dded25198e4d7d965fd9085e4ccd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.299430 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 8377b1aa704c43b78f310e7440f6488e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.299813 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8377b1aa704c43b78f310e7440f6488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.302582 31052 tablet_service.cc:1467] Processing CreateTablet for tablet f3dded25198e4d7d965fd9085e4ccd6d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.302980 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dded25198e4d7d965fd9085e4ccd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.304169 32436 tablet_bootstrap.cc:492] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:46.305806 32436 tablet_bootstrap.cc:654] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.306056 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:46.308142 32468 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:46.309743 32468 tablet_bootstrap.cc:654] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.326596 32466 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:46.327955 32482 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:46.328305 32466 tablet_bootstrap.cc:654] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.329581 32482 tablet_bootstrap.cc:654] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.332087 32468 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:46.332235 32468 ts_tablet_manager.cc:1367] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20240503 14:13:46.333056 32468 raft_consensus.cc:348] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.333276 32468 raft_consensus.cc:374] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.333345 32468 raft_consensus.cc:729] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.333557 32468 consensus_queue.cc:260] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.333673 30939 raft_consensus.cc:1261] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:46.334262 32468 ts_tablet_manager.cc:1398] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:46.334484 32453 consensus_queue.cc:1035] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.336699 32468 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:46.342375 32482 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:46.342527 32482 ts_tablet_manager.cc:1367] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240503 14:13:46.343418 32468 tablet_bootstrap.cc:654] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.343447 32482 raft_consensus.cc:348] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.343636 32482 raft_consensus.cc:374] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.343721 32482 raft_consensus.cc:729] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.343974 32482 consensus_queue.cc:260] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.344734 32482 ts_tablet_manager.cc:1398] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:46.344803 32436 tablet_bootstrap.cc:492] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:46.344954 32436 ts_tablet_manager.cc:1367] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.002s
I20240503 14:13:46.345930 32436 raft_consensus.cc:348] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.346148 32436 raft_consensus.cc:374] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.346262 32436 raft_consensus.cc:729] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.347757 32482 tablet_bootstrap.cc:492] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:46.347508 31070 raft_consensus.cc:1261] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:46.348387 32453 consensus_queue.cc:1035] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.348660 32436 consensus_queue.cc:260] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.349195 32436 ts_tablet_manager.cc:1398] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:46.349859 32482 tablet_bootstrap.cc:654] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.355433 32466 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:46.355682 32466 ts_tablet_manager.cc:1367] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20240503 14:13:46.356874 32466 raft_consensus.cc:348] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.359545 32466 raft_consensus.cc:374] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.360005 32466 raft_consensus.cc:729] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.361012 32466 consensus_queue.cc:260] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.364696 32466 ts_tablet_manager.cc:1398] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:46.374332 32483 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:46.376672 32483 tablet_bootstrap.cc:654] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.380566 32494 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:46.381176 32494 tablet.cc:1620] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:46.381511 27899 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 764c00a614f94fd48b90ada3b3ea7df1 (table rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280]). Expected version 1 got 0
I20240503 14:13:46.386853 32482 tablet_bootstrap.cc:492] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:46.387164 32482 ts_tablet_manager.cc:1367] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.016s
I20240503 14:13:46.388347 32482 raft_consensus.cc:348] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.388700 32482 raft_consensus.cc:374] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.388845 32482 raft_consensus.cc:729] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.389251 32482 consensus_queue.cc:260] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.390009 32482 ts_tablet_manager.cc:1398] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:46.392138 32468 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:46.392282 32468 ts_tablet_manager.cc:1367] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.005s
I20240503 14:13:46.397078 27898 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 764c00a614f94fd48b90ada3b3ea7df1 (table rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280]). Expected version 1 got 0
I20240503 14:13:46.399230 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280] alter complete (version 1)
I20240503 14:13:46.404321 32483 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:46.404546 32483 ts_tablet_manager.cc:1367] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.005s
I20240503 14:13:46.404621 32468 raft_consensus.cc:348] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.404881 32468 raft_consensus.cc:374] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.404995 32468 raft_consensus.cc:729] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.405246 32468 consensus_queue.cc:260] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.405582 32483 raft_consensus.cc:348] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.405848 32483 raft_consensus.cc:374] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.405977 32483 raft_consensus.cc:729] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.406260 32483 consensus_queue.cc:260] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.406663 32495 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:46.406972 32483 ts_tablet_manager.cc:1398] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:46.407312 32495 tablet.cc:1620] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:46.407588 32483 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:46.408979 32468 ts_tablet_manager.cc:1398] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.017s user 0.000s sys 0.003s
I20240503 14:13:46.409368 32468 tablet_bootstrap.cc:492] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:46.411352 27898 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 764c00a614f94fd48b90ada3b3ea7df1 (table rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280]). Expected version 1 got 0
I20240503 14:13:46.412214 32468 tablet_bootstrap.cc:654] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.420500 32483 tablet_bootstrap.cc:654] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.421887 32496 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:46.422657 32496 tablet.cc:1620] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:46.425267 27894 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 764c00a614f94fd48b90ada3b3ea7df1 (table rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280]). Expected version 1 got 0
I20240503 14:13:46.449949 32483 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:46.450147 32483 ts_tablet_manager.cc:1367] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.006s
I20240503 14:13:46.450776 32468 tablet_bootstrap.cc:492] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:46.450965 32468 ts_tablet_manager.cc:1367] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.028s
I20240503 14:13:46.450942 32483 raft_consensus.cc:348] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.451172 32483 raft_consensus.cc:374] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.451248 32483 raft_consensus.cc:729] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.451532 32483 consensus_queue.cc:260] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.452155 32483 ts_tablet_manager.cc:1398] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:46.452050 32468 raft_consensus.cc:348] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.452322 32468 raft_consensus.cc:374] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.452443 32468 raft_consensus.cc:729] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.452672 32468 consensus_queue.cc:260] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.454833 32468 ts_tablet_manager.cc:1398] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:46.490906 32475 raft_consensus.cc:482] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:46.491056 32475 raft_consensus.cc:504] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.491931 32475 leader_election.cc:288] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:46.492321 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dded25198e4d7d965fd9085e4ccd6d" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:46.492391 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dded25198e4d7d965fd9085e4ccd6d" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:46.492539 30810 raft_consensus.cc:2454] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:46.492576 31202 raft_consensus.cc:2454] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:46.492952 31006 leader_election.cc:302] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:46.493249 32475 raft_consensus.cc:2781] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:46.493356 32475 raft_consensus.cc:482] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:46.493449 32475 raft_consensus.cc:3037] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.495325 32475 raft_consensus.cc:504] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.495846 32475 leader_election.cc:288] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:46.496223 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dded25198e4d7d965fd9085e4ccd6d" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:46.496275 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dded25198e4d7d965fd9085e4ccd6d" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:46.496440 30810 raft_consensus.cc:3037] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.496450 31202 raft_consensus.cc:3037] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.499877 31202 raft_consensus.cc:2454] T f3dded25198e4d7d965fd9085e4ccd6d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:46.500329 31006 leader_election.cc:302] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:46.500646 32475 raft_consensus.cc:2781] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:46.500818 30810 raft_consensus.cc:2454] T f3dded25198e4d7d965fd9085e4ccd6d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:46.500856 32475 raft_consensus.cc:686] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:46.501799 32475 consensus_queue.cc:237] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:46.505123 27894 catalog_manager.cc:5484] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:46.511143 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:46.518116 32494 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:46.518627 32495 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:46.518680 32494 tablet.cc:1620] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:46.519014 32495 tablet.cc:1620] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:46.519706 32496 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:46.520097 32496 tablet.cc:1620] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:46.528663 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=dffe3d05e30842b8bba875220ae55280] alter complete (version 2)
I20240503 14:13:46.544641 32125 raft_consensus.cc:482] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:46.544870 32125 raft_consensus.cc:504] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.545579 32125 leader_election.cc:288] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:46.545913 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377b1aa704c43b78f310e7440f6488e" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:46.546085 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377b1aa704c43b78f310e7440f6488e" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:46.546170 30940 raft_consensus.cc:2454] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:46.546397 31070 raft_consensus.cc:2454] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:46.546828 28239 leader_election.cc:302] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:46.547125 32125 raft_consensus.cc:2781] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:46.547217 32125 raft_consensus.cc:482] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:46.547324 32125 raft_consensus.cc:3037] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.549252 32125 raft_consensus.cc:504] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.549752 32125 leader_election.cc:288] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:46.550035 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377b1aa704c43b78f310e7440f6488e" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:46.550227 30940 raft_consensus.cc:3037] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.550180 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8377b1aa704c43b78f310e7440f6488e" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:46.550392 31070 raft_consensus.cc:3037] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.552037 30940 raft_consensus.cc:2454] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:46.552174 31070 raft_consensus.cc:2454] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:46.552466 28239 leader_election.cc:302] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:46.552767 32125 raft_consensus.cc:2781] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:46.552946 32125 raft_consensus.cc:686] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:46.553231 32125 consensus_queue.cc:237] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:46.557178 27900 catalog_manager.cc:5484] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:46.631115 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:46.640338 32494 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:46.643275 32496 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:46.644672 32495 tablet.cc:1722] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:46.654529 27900 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=dffe3d05e30842b8bba875220ae55280] alter complete (version 3)
I20240503 14:13:46.713326 32475 raft_consensus.cc:482] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:46.713524 32475 raft_consensus.cc:504] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.714280 32475 leader_election.cc:288] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:46.714771 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6b6e0cac9a4ea6a0189970d88f9d72" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:46.714771 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6b6e0cac9a4ea6a0189970d88f9d72" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:46.714998 28304 raft_consensus.cc:2454] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:46.715011 30809 raft_consensus.cc:2454] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:46.715361 31005 leader_election.cc:302] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:46.715655 32475 raft_consensus.cc:2781] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:46.715759 32475 raft_consensus.cc:482] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:46.715853 32475 raft_consensus.cc:3037] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.717823 32475 raft_consensus.cc:504] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.718386 32475 leader_election.cc:288] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:46.718883 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6b6e0cac9a4ea6a0189970d88f9d72" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:46.718890 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b6b6e0cac9a4ea6a0189970d88f9d72" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:46.719077 30809 raft_consensus.cc:3037] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.719069 28304 raft_consensus.cc:3037] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.721683 30809 raft_consensus.cc:2454] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:46.721794 28304 raft_consensus.cc:2454] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:46.722169 31005 leader_election.cc:302] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:46.722481 32475 raft_consensus.cc:2781] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:46.722635 32475 raft_consensus.cc:686] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:46.722909 32475 consensus_queue.cc:237] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:46.726714 27894 catalog_manager.cc:5484] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:46.757189 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:46.757412 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:46.760389 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 764c00a614f94fd48b90ada3b3ea7df1
I20240503 14:13:46.761312 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 764c00a614f94fd48b90ada3b3ea7df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:46 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:46.761375 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 764c00a614f94fd48b90ada3b3ea7df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:46 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:46.761395 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 764c00a614f94fd48b90ada3b3ea7df1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:46 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:46.762001 32502 tablet_replica.cc:330] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:46.762185 32504 tablet_replica.cc:330] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:46.762215 32502 raft_consensus.cc:2229] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:46.762466 32502 raft_consensus.cc:2258] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:46.762573 32504 raft_consensus.cc:2229] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:46.763214 32504 raft_consensus.cc:2258] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:46.763185 32503 tablet_replica.cc:330] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:46.763463 32503 raft_consensus.cc:2229] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:46.764127 32504 ts_tablet_manager.cc:1875] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:46.764526 32502 ts_tablet_manager.cc:1875] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:46.768174 32503 raft_consensus.cc:2258] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:46.769068 32503 ts_tablet_manager.cc:1875] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:46.771031 32504 ts_tablet_manager.cc:1888] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:46.771136 32504 log.cc:1196] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/764c00a614f94fd48b90ada3b3ea7df1
I20240503 14:13:46.771526 32504 ts_tablet_manager.cc:1909] T 764c00a614f94fd48b90ada3b3ea7df1 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:46.772356 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 764c00a614f94fd48b90ada3b3ea7df1 (table rebalancer_dynamic_table_new_name [id=dffe3d05e30842b8bba875220ae55280]) successfully deleted
I20240503 14:13:46.773000 32502 ts_tablet_manager.cc:1888] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:46.773114 32502 log.cc:1196] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/764c00a614f94fd48b90ada3b3ea7df1
I20240503 14:13:46.773581 32502 ts_tablet_manager.cc:1909] T 764c00a614f94fd48b90ada3b3ea7df1 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:46.774603 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 764c00a614f94fd48b90ada3b3ea7df1 (table rebalancer_dynamic_table_new_name [id=dffe3d05e30842b8bba875220ae55280]) successfully deleted
I20240503 14:13:46.775456 32503 ts_tablet_manager.cc:1888] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:46.775557 32503 log.cc:1196] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/764c00a614f94fd48b90ada3b3ea7df1
I20240503 14:13:46.776206 32503 ts_tablet_manager.cc:1909] T 764c00a614f94fd48b90ada3b3ea7df1 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:46.777315 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 764c00a614f94fd48b90ada3b3ea7df1 (table rebalancer_dynamic_table_new_name [id=dffe3d05e30842b8bba875220ae55280]) successfully deleted
I20240503 14:13:46.863111 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:46.876763 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 7aeb18146a544c8da8776f95a82c5749 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41266ab12241419b962cc9221f87fb52]), partition=
I20240503 14:13:46.876844 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 7aeb18146a544c8da8776f95a82c5749 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41266ab12241419b962cc9221f87fb52]), partition=
I20240503 14:13:46.877128 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aeb18146a544c8da8776f95a82c5749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.877216 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aeb18146a544c8da8776f95a82c5749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.883401 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 7aeb18146a544c8da8776f95a82c5749 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=41266ab12241419b962cc9221f87fb52]), partition=
I20240503 14:13:46.883760 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aeb18146a544c8da8776f95a82c5749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.902686 32506 tablet_bootstrap.cc:492] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:46.903856 32505 tablet_bootstrap.cc:492] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:46.904729 32506 tablet_bootstrap.cc:654] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.905018 32505 tablet_bootstrap.cc:654] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.906448 32507 tablet_bootstrap.cc:492] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:46.908216 32507 tablet_bootstrap.cc:654] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.908710 32506 tablet_bootstrap.cc:492] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:46.908872 32506 ts_tablet_manager.cc:1367] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:46.910017 32506 raft_consensus.cc:348] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:46.910558 32506 raft_consensus.cc:374] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.910640 32505 tablet_bootstrap.cc:492] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:46.910712 32506 raft_consensus.cc:729] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.911094 32505 ts_tablet_manager.cc:1367] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:46.911365 32506 consensus_queue.cc:260] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:46.912479 32505 raft_consensus.cc:348] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:46.912716 32505 raft_consensus.cc:374] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.912823 32505 raft_consensus.cc:729] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.913287 32505 consensus_queue.cc:260] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:46.913749 32507 tablet_bootstrap.cc:492] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:46.913914 32507 ts_tablet_manager.cc:1367] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:13:46.914772 32505 ts_tablet_manager.cc:1398] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:46.914710 32507 raft_consensus.cc:348] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:46.914937 32507 raft_consensus.cc:374] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.915051 32507 raft_consensus.cc:729] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.915270 32507 consensus_queue.cc:260] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:46.915815 32507 ts_tablet_manager.cc:1398] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:46.922426 32506 ts_tablet_manager.cc:1398] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:46.925210 32475 consensus_queue.cc:1035] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.930765 32475 consensus_queue.cc:1035] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:46.937165 32125 consensus_queue.cc:1035] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.941879 32376 consensus_queue.cc:1035] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.048759 32438 raft_consensus.cc:482] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.048949 32438 raft_consensus.cc:504] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.049620 32438 leader_election.cc:288] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:47.050045 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeb18146a544c8da8776f95a82c5749" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:47.050302 31203 raft_consensus.cc:2454] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:47.050796 30875 leader_election.cc:302] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:47.050832 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeb18146a544c8da8776f95a82c5749" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:47.051069 28434 raft_consensus.cc:2454] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:47.051117 32438 raft_consensus.cc:2781] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.051215 32438 raft_consensus.cc:482] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.051299 32438 raft_consensus.cc:3037] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.053462 32438 raft_consensus.cc:504] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.054131 32438 leader_election.cc:288] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:47.054656 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeb18146a544c8da8776f95a82c5749" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:47.054832 31203 raft_consensus.cc:3037] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.056849 31203 raft_consensus.cc:2454] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:47.057714 30875 leader_election.cc:302] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:47.058043 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aeb18146a544c8da8776f95a82c5749" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:47.058157 32190 raft_consensus.cc:2781] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.058312 32190 raft_consensus.cc:686] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:47.058532 28434 raft_consensus.cc:3037] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.058655 32190 consensus_queue.cc:237] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.062551 28434 raft_consensus.cc:2454] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:47.071049 27898 catalog_manager.cc:5484] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:47.137086 32475 consensus_queue.cc:1035] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.141438 32476 consensus_queue.cc:1035] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.186188 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:47.191824 31202 raft_consensus.cc:1261] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:47.191843 28434 raft_consensus.cc:1261] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:47.192366 32517 consensus_queue.cc:1035] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.192616 32190 consensus_queue.cc:1035] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.198341 32488 mvcc.cc:205] Tried to move back new op lower bound from 7023598088971935744 to 7023598088435527680. Current Snapshot: MvccSnapshot[applied={T|T < 7023598088971935744}]
I20240503 14:13:47.198693 32523 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:47.199457 32523 tablet.cc:1620] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:47.201187 32524 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:47.202606 32524 tablet.cc:1620] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:47.203452 27898 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 7aeb18146a544c8da8776f95a82c5749 (table rebalancer_dynamic_table [id=41266ab12241419b962cc9221f87fb52]). Expected version 1 got 0
I20240503 14:13:47.210695 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=41266ab12241419b962cc9221f87fb52] alter complete (version 1)
I20240503 14:13:47.211865 27894 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 7aeb18146a544c8da8776f95a82c5749 (table rebalancer_dynamic_table [id=41266ab12241419b962cc9221f87fb52]). Expected version 1 got 0
I20240503 14:13:47.213156 32526 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:47.213793 32526 tablet.cc:1620] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:47.265053 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:47.276556 30790 tablet_service.cc:1467] Processing CreateTablet for tablet d0d2600fd84a4d839dc3366b08485e42 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.276715 28154 tablet_service.cc:1467] Processing CreateTablet for tablet d0d2600fd84a4d839dc3366b08485e42 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.277060 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d2600fd84a4d839dc3366b08485e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.277117 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d2600fd84a4d839dc3366b08485e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.277086 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 6225866853d94dda88c87c6af053c739 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.277487 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6225866853d94dda88c87c6af053c739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.280659 30920 tablet_service.cc:1467] Processing CreateTablet for tablet d0d2600fd84a4d839dc3366b08485e42 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.281052 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d2600fd84a4d839dc3366b08485e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.284086 32527 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:47.284878 32528 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:47.286293 32527 tablet_bootstrap.cc:654] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.286803 32528 tablet_bootstrap.cc:654] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.289894 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 6225866853d94dda88c87c6af053c739 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.290309 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6225866853d94dda88c87c6af053c739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.291198 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 6225866853d94dda88c87c6af053c739 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.291594 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6225866853d94dda88c87c6af053c739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.291954 30921 tablet_service.cc:1467] Processing CreateTablet for tablet e571695c819e485483ece09278ee1e09 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.292368 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e571695c819e485483ece09278ee1e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.297286 32505 tablet_bootstrap.cc:492] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:47.298597 32507 tablet_bootstrap.cc:492] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:47.298909 32505 tablet_bootstrap.cc:654] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.300704 32506 tablet_bootstrap.cc:492] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:47.300999 32507 tablet_bootstrap.cc:654] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.302078 31051 tablet_service.cc:1467] Processing CreateTablet for tablet e571695c819e485483ece09278ee1e09 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.302330 32506 tablet_bootstrap.cc:654] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.302556 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e571695c819e485483ece09278ee1e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.303625 32528 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:47.304034 32528 ts_tablet_manager.cc:1367] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20240503 14:13:47.305061 32528 raft_consensus.cc:348] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.307089 32528 raft_consensus.cc:374] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.307375 32528 raft_consensus.cc:729] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.307824 32528 consensus_queue.cc:260] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.308984 32528 ts_tablet_manager.cc:1398] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:47.313230 28154 tablet_service.cc:1467] Processing CreateTablet for tablet e571695c819e485483ece09278ee1e09 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.313731 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e571695c819e485483ece09278ee1e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.315589 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:47.318598 32534 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:47.320209 32534 tablet_bootstrap.cc:654] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.322804 32528 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:47.324381 32506 tablet_bootstrap.cc:492] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:47.324419 32528 tablet_bootstrap.cc:654] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.324554 32506 ts_tablet_manager.cc:1367] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.008s
I20240503 14:13:47.325343 32506 raft_consensus.cc:348] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.325580 32506 raft_consensus.cc:374] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.325677 32506 raft_consensus.cc:729] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.326103 32506 consensus_queue.cc:260] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.326669 32506 ts_tablet_manager.cc:1398] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:47.327529 32523 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:47.328147 32523 tablet.cc:1620] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:47.332428 32526 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:47.332863 32526 tablet.cc:1620] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:47.336858 32507 tablet_bootstrap.cc:492] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:47.337082 32507 ts_tablet_manager.cc:1367] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.005s
I20240503 14:13:47.337303 32453 raft_consensus.cc:482] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.337471 32453 raft_consensus.cc:504] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.338220 32453 leader_election.cc:288] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:47.340006 32524 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:47.343984 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:47.344538 32524 tablet.cc:1620] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
W20240503 14:13:47.344630 31139 leader_election.cc:341] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:47.343327 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
W20240503 14:13:47.345404 31138 leader_election.cc:341] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:47.345520 31138 leader_election.cc:302] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:13:47.345979 32453 raft_consensus.cc:2726] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:47.347592 32507 raft_consensus.cc:348] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.347971 32507 raft_consensus.cc:374] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.348088 32507 raft_consensus.cc:729] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.348369 32507 consensus_queue.cc:260] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.349210 32507 ts_tablet_manager.cc:1398] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:47.349534 32507 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:47.351629 32190 raft_consensus.cc:482] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.351763 32190 raft_consensus.cc:504] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.352433 32190 leader_election.cc:288] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:47.353816 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
W20240503 14:13:47.354351 30877 leader_election.cc:341] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:47.354602 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:47.354924 31202 raft_consensus.cc:2454] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:47.355319 32507 tablet_bootstrap.cc:654] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.355517 30875 leader_election.cc:302] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters: 5388902973134958a030fd63ab4faba4
I20240503 14:13:47.356117 32190 raft_consensus.cc:2781] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.356274 32190 raft_consensus.cc:482] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.356426 32190 raft_consensus.cc:3037] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.359117 32190 raft_consensus.cc:504] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.360316 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:47.360352 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=41266ab12241419b962cc9221f87fb52] alter complete (version 2)
I20240503 14:13:47.360466 32505 tablet_bootstrap.cc:492] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:47.360500 31202 raft_consensus.cc:3037] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.360648 32505 ts_tablet_manager.cc:1367] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.005s
I20240503 14:13:47.361457 32505 raft_consensus.cc:348] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.361680 32505 raft_consensus.cc:374] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.361752 32505 raft_consensus.cc:729] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.361963 32505 consensus_queue.cc:260] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.363209 31202 raft_consensus.cc:2454] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:47.363869 30875 leader_election.cc:302] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:47.364158 32527 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:47.364295 32527 ts_tablet_manager.cc:1367] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.080s user 0.017s sys 0.022s
I20240503 14:13:47.364368 32517 raft_consensus.cc:2781] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.364554 32517 raft_consensus.cc:686] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:47.364817 32517 consensus_queue.cc:237] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:47.365173 32527 raft_consensus.cc:348] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.365309 32505 ts_tablet_manager.cc:1398] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 14:13:47.365368 32527 raft_consensus.cc:374] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.365470 32527 raft_consensus.cc:729] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.365770 32527 consensus_queue.cc:260] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.366343 32527 ts_tablet_manager.cc:1398] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:47.368256 32528 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:47.368453 32528 ts_tablet_manager.cc:1367] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
I20240503 14:13:47.369493 32528 raft_consensus.cc:348] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.369781 32528 raft_consensus.cc:374] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.369874 32528 raft_consensus.cc:729] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.370065 32426 raft_consensus.cc:482] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.370276 32426 raft_consensus.cc:504] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.370213 32528 consensus_queue.cc:260] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.370976 32528 ts_tablet_manager.cc:1398] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:47.371080 32426 leader_election.cc:288] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:47.371830 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d2600fd84a4d839dc3366b08485e42" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:47.372046 30809 raft_consensus.cc:2454] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:47.372526 28107 leader_election.cc:302] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:47.372809 32345 raft_consensus.cc:2781] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.372920 32345 raft_consensus.cc:482] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.373005 32345 raft_consensus.cc:3037] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.374769 32190 leader_election.cc:288] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:47.375684 32345 raft_consensus.cc:504] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.377100 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d2600fd84a4d839dc3366b08485e42" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:47.377152 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d2600fd84a4d839dc3366b08485e42" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:47.377611 32345 leader_election.cc:288] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:47.378350 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d2600fd84a4d839dc3366b08485e42" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
W20240503 14:13:47.378563 28109 leader_election.cc:341] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:47.378578 30809 raft_consensus.cc:3037] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:47.378881 28109 leader_election.cc:341] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:47.379612 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:47.379856 28432 raft_consensus.cc:3037] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.381903 30809 raft_consensus.cc:2454] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:47.382500 28107 leader_election.cc:302] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:13:47.382787 32426 raft_consensus.cc:2781] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.382927 32426 raft_consensus.cc:686] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:47.383210 32426 consensus_queue.cc:237] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.384511 28432 raft_consensus.cc:2454] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:47.388722 27898 catalog_manager.cc:5484] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:47.391520 27900 catalog_manager.cc:5484] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:47.393677 32534 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:47.393833 32534 ts_tablet_manager.cc:1367] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.003s
I20240503 14:13:47.394743 32534 raft_consensus.cc:348] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.394949 32534 raft_consensus.cc:374] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.395044 32534 raft_consensus.cc:729] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.395287 32534 consensus_queue.cc:260] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.395836 32534 ts_tablet_manager.cc:1398] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:47.409291 32475 raft_consensus.cc:482] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.409556 32507 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:47.409487 32475 raft_consensus.cc:504] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.409832 32507 ts_tablet_manager.cc:1367] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.060s user 0.013s sys 0.006s
I20240503 14:13:47.410399 32475 leader_election.cc:288] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:47.410885 32507 raft_consensus.cc:348] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.411177 32507 raft_consensus.cc:374] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.411376 32507 raft_consensus.cc:729] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.411716 32507 consensus_queue.cc:260] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.411775 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e571695c819e485483ece09278ee1e09" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:47.412025 28174 raft_consensus.cc:2454] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:47.412536 32507 ts_tablet_manager.cc:1398] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:47.412892 32507 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:47.413039 31006 leader_election.cc:302] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:47.413570 32476 raft_consensus.cc:2781] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.413684 32476 raft_consensus.cc:482] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.413787 32476 raft_consensus.cc:3037] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.414865 32507 tablet_bootstrap.cc:654] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.411176 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e571695c819e485483ece09278ee1e09" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:47.415184 30940 raft_consensus.cc:2454] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:47.423851 32476 raft_consensus.cc:504] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:47.426759 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e571695c819e485483ece09278ee1e09" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:47.427050 30940 raft_consensus.cc:3037] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.436662 32476 leader_election.cc:288] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:47.436309 30940 raft_consensus.cc:2454] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:47.437878 31007 leader_election.cc:302] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:47.438287 32476 raft_consensus.cc:2781] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.438448 32476 raft_consensus.cc:686] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:47.438943 32476 consensus_queue.cc:237] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:13:47.463052 31006 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.72:41331, user_credentials={real_user=slave}} blocked reactor thread for 44490.9us
I20240503 14:13:47.463997 32507 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:47.463933 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e571695c819e485483ece09278ee1e09" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:47.464159 28174 raft_consensus.cc:3037] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.464172 32507 ts_tablet_manager.cc:1367] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.051s user 0.027s sys 0.018s
I20240503 14:13:47.465037 32507 raft_consensus.cc:348] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.465266 32507 raft_consensus.cc:374] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.465342 32507 raft_consensus.cc:729] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.465576 32507 consensus_queue.cc:260] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.466213 32507 ts_tablet_manager.cc:1398] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:13:47.466794 28174 raft_consensus.cc:2454] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:47.470445 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:47.471300 27898 catalog_manager.cc:5484] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:47.479310 32523 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:47.480744 32526 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:47.487840 32524 tablet.cc:1722] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:47.495201 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=41266ab12241419b962cc9221f87fb52] alter complete (version 3)
I20240503 14:13:47.531361 32190 raft_consensus.cc:482] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.531553 32190 raft_consensus.cc:504] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:47.532338 32190 leader_election.cc:288] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:47.532994 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d2600fd84a4d839dc3366b08485e42" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:47.533066 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0d2600fd84a4d839dc3366b08485e42" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:47.533284 30809 raft_consensus.cc:2379] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bbc686598f24f518d7ca6bb2063058d in current term 1: Already voted for candidate 6409961ee8c0445687c11724213c44a9 in this term.
I20240503 14:13:47.533685 30874 leader_election.cc:302] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d; no voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:13:47.533918 32190 raft_consensus.cc:3037] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.536280 32190 raft_consensus.cc:2726] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:47.597967 27900 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:47.598853 27900 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:47.601924 27900 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 7aeb18146a544c8da8776f95a82c5749
I20240503 14:13:47.602659 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aeb18146a544c8da8776f95a82c5749 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:47 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:47.602874 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aeb18146a544c8da8776f95a82c5749 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:47 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:47.603083 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aeb18146a544c8da8776f95a82c5749 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:47 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:47.603701 32546 tablet_replica.cc:330] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:47.603920 32545 tablet_replica.cc:330] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:47.604104 32546 raft_consensus.cc:2229] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:47.604233 32545 raft_consensus.cc:2229] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:47.604498 32545 raft_consensus.cc:2258] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.604632 32547 tablet_replica.cc:330] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:47.605010 32547 raft_consensus.cc:2229] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:47.605264 32547 raft_consensus.cc:2258] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.605609 32545 ts_tablet_manager.cc:1875] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:47.605876 32546 raft_consensus.cc:2258] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.606874 32547 ts_tablet_manager.cc:1875] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:47.611856 32546 ts_tablet_manager.cc:1875] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:47.613013 32545 ts_tablet_manager.cc:1888] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:47.613181 32545 log.cc:1196] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/7aeb18146a544c8da8776f95a82c5749
I20240503 14:13:47.613519 32547 ts_tablet_manager.cc:1888] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:47.613729 32545 ts_tablet_manager.cc:1909] T 7aeb18146a544c8da8776f95a82c5749 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:47.613729 32547 log.cc:1196] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/7aeb18146a544c8da8776f95a82c5749
I20240503 14:13:47.614519 32547 ts_tablet_manager.cc:1909] T 7aeb18146a544c8da8776f95a82c5749 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:47.614956 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 7aeb18146a544c8da8776f95a82c5749 (table rebalancer_dynamic_table_new_name [id=41266ab12241419b962cc9221f87fb52]) successfully deleted
I20240503 14:13:47.615700 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 7aeb18146a544c8da8776f95a82c5749 (table rebalancer_dynamic_table_new_name [id=41266ab12241419b962cc9221f87fb52]) successfully deleted
I20240503 14:13:47.619639 32546 ts_tablet_manager.cc:1888] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:47.619823 32546 log.cc:1196] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/7aeb18146a544c8da8776f95a82c5749
I20240503 14:13:47.620272 32546 ts_tablet_manager.cc:1909] T 7aeb18146a544c8da8776f95a82c5749 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:47.621205 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 7aeb18146a544c8da8776f95a82c5749 (table rebalancer_dynamic_table_new_name [id=41266ab12241419b962cc9221f87fb52]) successfully deleted
I20240503 14:13:47.704226 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:47.711515 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 6b25f3678e0c433e948ac0bb7e4a618d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb88d0f4f9e3447da11754a4bb6a75d5]), partition=
I20240503 14:13:47.711764 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 6b25f3678e0c433e948ac0bb7e4a618d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb88d0f4f9e3447da11754a4bb6a75d5]), partition=
I20240503 14:13:47.711898 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b25f3678e0c433e948ac0bb7e4a618d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.712081 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 6b25f3678e0c433e948ac0bb7e4a618d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=eb88d0f4f9e3447da11754a4bb6a75d5]), partition=
I20240503 14:13:47.712144 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b25f3678e0c433e948ac0bb7e4a618d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.712419 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b25f3678e0c433e948ac0bb7e4a618d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.717350 32507 tablet_bootstrap.cc:492] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:47.718639 32507 tablet_bootstrap.cc:654] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.719020 32549 tablet_bootstrap.cc:492] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:47.719245 32527 tablet_bootstrap.cc:492] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:47.720813 32549 tablet_bootstrap.cc:654] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.720891 32527 tablet_bootstrap.cc:654] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.721644 32507 tablet_bootstrap.cc:492] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:47.721822 32507 ts_tablet_manager.cc:1367] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:13:47.722901 32507 raft_consensus.cc:348] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:47.723270 32507 raft_consensus.cc:374] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.723395 32507 raft_consensus.cc:729] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.723675 32507 consensus_queue.cc:260] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:47.724419 32507 ts_tablet_manager.cc:1398] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:47.725704 32527 tablet_bootstrap.cc:492] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:47.725864 32527 ts_tablet_manager.cc:1367] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 14:13:47.727488 32549 tablet_bootstrap.cc:492] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:47.727716 32549 ts_tablet_manager.cc:1367] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 14:13:47.728243 32527 raft_consensus.cc:348] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:47.728469 32527 raft_consensus.cc:374] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.728550 32527 raft_consensus.cc:729] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.728502 32549 raft_consensus.cc:348] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:47.728727 32549 raft_consensus.cc:374] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.728827 32549 raft_consensus.cc:729] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.728787 32527 consensus_queue.cc:260] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:47.729307 32527 ts_tablet_manager.cc:1398] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:47.729075 32549 consensus_queue.cc:260] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:47.730731 32549 ts_tablet_manager.cc:1398] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:47.779574 32426 consensus_queue.cc:1035] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.784179 32426 consensus_queue.cc:1035] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.886600 32190 consensus_queue.cc:1035] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.888015 32475 consensus_queue.cc:1035] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.897563 32475 consensus_queue.cc:1035] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.909907 32517 consensus_queue.cc:1035] T 6225866853d94dda88c87c6af053c739 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.991479 27900 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:48.000306 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 1058a6f8db004b50a51a4c06152c7756 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.000353 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 1058a6f8db004b50a51a4c06152c7756 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.000762 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1058a6f8db004b50a51a4c06152c7756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.000775 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1058a6f8db004b50a51a4c06152c7756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.001567 28154 tablet_service.cc:1467] Processing CreateTablet for tablet dcb14382755441afb4cc5c103425331c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.001946 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb14382755441afb4cc5c103425331c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.003124 31052 tablet_service.cc:1467] Processing CreateTablet for tablet dcb14382755441afb4cc5c103425331c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.003293 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 6b263006614c46b4ac0648efb7a37a8f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.003556 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb14382755441afb4cc5c103425331c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.003688 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b263006614c46b4ac0648efb7a37a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.005872 30921 tablet_service.cc:1467] Processing CreateTablet for tablet dcb14382755441afb4cc5c103425331c (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.006258 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb14382755441afb4cc5c103425331c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.007227 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 6b263006614c46b4ac0648efb7a37a8f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.007583 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b263006614c46b4ac0648efb7a37a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.011408 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 6b263006614c46b4ac0648efb7a37a8f (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.011848 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b263006614c46b4ac0648efb7a37a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.013808 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 1058a6f8db004b50a51a4c06152c7756 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.015097 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1058a6f8db004b50a51a4c06152c7756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.018136 32507 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:48.020042 32507 tablet_bootstrap.cc:654] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.024060 32400 raft_consensus.cc:482] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:48.024273 32400 raft_consensus.cc:504] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.025395 32566 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:48.025676 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b25f3678e0c433e948ac0bb7e4a618d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:48.025913 28304 raft_consensus.cc:2454] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:48.026469 30746 leader_election.cc:302] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:48.027007 32325 raft_consensus.cc:2781] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:48.027128 32325 raft_consensus.cc:482] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:48.027215 32400 leader_election.cc:288] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:48.027238 32325 raft_consensus.cc:3037] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.027879 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b25f3678e0c433e948ac0bb7e4a618d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:48.028107 30940 raft_consensus.cc:2454] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:48.028860 32566 tablet_bootstrap.cc:654] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.030810 32507 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:48.031021 32507 ts_tablet_manager.cc:1367] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240503 14:13:48.031230 32325 raft_consensus.cc:504] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.031857 32507 raft_consensus.cc:348] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.032074 32507 raft_consensus.cc:374] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.032163 32507 raft_consensus.cc:729] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.032420 32507 consensus_queue.cc:260] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.032635 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b25f3678e0c433e948ac0bb7e4a618d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:48.032861 28304 raft_consensus.cc:3037] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.034286 32507 ts_tablet_manager.cc:1398] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:48.035738 28304 raft_consensus.cc:2454] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:48.036201 32325 leader_election.cc:288] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:48.036792 32571 tablet_bootstrap.cc:492] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:48.039230 32571 tablet_bootstrap.cc:654] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.039691 32527 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:48.040452 32507 tablet_bootstrap.cc:492] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:48.041291 32527 tablet_bootstrap.cc:654] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.042002 32507 tablet_bootstrap.cc:654] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.050446 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b25f3678e0c433e948ac0bb7e4a618d" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:48.050849 30940 raft_consensus.cc:3037] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.057494 30940 raft_consensus.cc:2454] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:48.059273 30745 leader_election.cc:302] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:48.059588 32569 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:48.059849 32400 raft_consensus.cc:2781] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:48.060132 32400 raft_consensus.cc:686] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:48.060667 32400 consensus_queue.cc:237] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.064458 32569 tablet_bootstrap.cc:654] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.064885 32567 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:48.066892 32507 tablet_bootstrap.cc:492] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:48.067130 32507 ts_tablet_manager.cc:1367] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.006s
I20240503 14:13:48.068097 32507 raft_consensus.cc:348] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.068310 32527 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:48.068405 32507 raft_consensus.cc:374] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.068550 32527 ts_tablet_manager.cc:1367] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.006s
I20240503 14:13:48.068562 32507 raft_consensus.cc:729] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.069306 32527 raft_consensus.cc:348] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.069505 32527 raft_consensus.cc:374] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.069595 32527 raft_consensus.cc:729] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.068902 32507 consensus_queue.cc:260] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.070915 32527 consensus_queue.cc:260] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.071450 32527 ts_tablet_manager.cc:1398] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:48.072778 32507 ts_tablet_manager.cc:1398] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:48.075204 32567 tablet_bootstrap.cc:654] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.081094 32571 tablet_bootstrap.cc:492] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:48.081401 32571 ts_tablet_manager.cc:1367] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.029s
I20240503 14:13:48.081032 27899 catalog_manager.cc:5484] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:48.082525 32571 raft_consensus.cc:348] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.082816 32571 raft_consensus.cc:374] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.082950 32571 raft_consensus.cc:729] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.083247 32571 consensus_queue.cc:260] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.086611 32571 ts_tablet_manager.cc:1398] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:48.094062 32567 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:48.094302 32567 ts_tablet_manager.cc:1367] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.005s
I20240503 14:13:48.095047 32567 raft_consensus.cc:348] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.095278 32567 raft_consensus.cc:374] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.095925 32567 raft_consensus.cc:729] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.095995 32566 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:48.096143 32566 ts_tablet_manager.cc:1367] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.006s
I20240503 14:13:48.096235 32567 consensus_queue.cc:260] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.096968 32566 raft_consensus.cc:348] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.097184 32566 raft_consensus.cc:374] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.097321 32566 raft_consensus.cc:729] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.097559 32566 consensus_queue.cc:260] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.098101 32566 ts_tablet_manager.cc:1398] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:48.099915 32566 tablet_bootstrap.cc:492] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:48.101656 32566 tablet_bootstrap.cc:654] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.108160 32569 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:48.108479 32569 ts_tablet_manager.cc:1367] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.000s
I20240503 14:13:48.110150 32569 raft_consensus.cc:348] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.111527 32569 raft_consensus.cc:374] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.111728 32569 raft_consensus.cc:729] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.112587 32569 consensus_queue.cc:260] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.113497 32569 ts_tablet_manager.cc:1398] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:48.114173 32567 ts_tablet_manager.cc:1398] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20240503 14:13:48.124586 32566 tablet_bootstrap.cc:492] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:48.124768 32566 ts_tablet_manager.cc:1367] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.009s
I20240503 14:13:48.125595 32566 raft_consensus.cc:348] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.125761 32566 raft_consensus.cc:374] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.125820 32566 raft_consensus.cc:729] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.126024 32566 consensus_queue.cc:260] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.126533 32566 ts_tablet_manager.cc:1398] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:48.127033 32566 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:48.128418 32566 tablet_bootstrap.cc:654] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.130223 32566 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:48.130513 32566 ts_tablet_manager.cc:1367] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:48.131265 32566 raft_consensus.cc:348] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.131498 32566 raft_consensus.cc:374] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.131606 32566 raft_consensus.cc:729] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.131853 32566 consensus_queue.cc:260] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.132349 32566 ts_tablet_manager.cc:1398] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:48.147925 32400 raft_consensus.cc:482] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:48.148130 32400 raft_consensus.cc:504] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.148783 32400 leader_election.cc:288] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:48.149166 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1058a6f8db004b50a51a4c06152c7756" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:48.149250 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1058a6f8db004b50a51a4c06152c7756" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:48.149389 31070 raft_consensus.cc:2454] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:48.149477 28434 raft_consensus.cc:2454] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:48.149827 30745 leader_election.cc:302] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:48.150120 32400 raft_consensus.cc:2781] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:48.150331 32400 raft_consensus.cc:482] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:48.150460 32400 raft_consensus.cc:3037] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.152276 32400 raft_consensus.cc:504] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.152786 32400 leader_election.cc:288] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:48.153157 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1058a6f8db004b50a51a4c06152c7756" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:48.153184 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1058a6f8db004b50a51a4c06152c7756" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:48.153347 31070 raft_consensus.cc:3037] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.153342 28434 raft_consensus.cc:3037] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.155339 31070 raft_consensus.cc:2454] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:48.155758 30745 leader_election.cc:302] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:48.155905 28434 raft_consensus.cc:2454] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:48.156078 32400 raft_consensus.cc:2781] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:48.156271 32400 raft_consensus.cc:686] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:48.156540 32400 consensus_queue.cc:237] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:48.160044 27899 catalog_manager.cc:5484] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:48.210685 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:48.216004 30940 raft_consensus.cc:1261] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:48.216281 28304 raft_consensus.cc:1261] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:48.216578 32400 consensus_queue.cc:1035] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.216837 32570 consensus_queue.cc:1035] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.226876 27899 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 6b25f3678e0c433e948ac0bb7e4a618d (table rebalancer_dynamic_table [id=eb88d0f4f9e3447da11754a4bb6a75d5]). Expected version 1 got 0
I20240503 14:13:48.229430 32586 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:48.230020 32586 tablet.cc:1620] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:48.231552 32585 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:48.233091 32585 tablet.cc:1620] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:48.237771 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=eb88d0f4f9e3447da11754a4bb6a75d5] alter complete (version 1)
I20240503 14:13:48.240916 27899 catalog_manager.cc:5576] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889) does not have the latest schema for tablet 6b25f3678e0c433e948ac0bb7e4a618d (table rebalancer_dynamic_table [id=eb88d0f4f9e3447da11754a4bb6a75d5]). Expected version 1 got 0
I20240503 14:13:48.241539 32588 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:48.242165 32588 tablet.cc:1620] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:48.312177 32345 raft_consensus.cc:482] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:48.312831 32345 raft_consensus.cc:504] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.313472 32345 leader_election.cc:288] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:48.313870 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb14382755441afb4cc5c103425331c" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:48.313993 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb14382755441afb4cc5c103425331c" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:48.314164 31072 raft_consensus.cc:2454] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:48.314262 30940 raft_consensus.cc:2454] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:48.314643 28109 leader_election.cc:302] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:48.314924 32345 raft_consensus.cc:2781] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:48.315061 32345 raft_consensus.cc:482] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:48.315191 32345 raft_consensus.cc:3037] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.317195 32345 raft_consensus.cc:504] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.317592 32345 leader_election.cc:288] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:48.317986 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb14382755441afb4cc5c103425331c" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:48.318059 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb14382755441afb4cc5c103425331c" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:48.318179 31072 raft_consensus.cc:3037] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.318224 30940 raft_consensus.cc:3037] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.320057 31072 raft_consensus.cc:2454] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:48.320078 30940 raft_consensus.cc:2454] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:48.320405 28109 leader_election.cc:302] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:48.320658 32345 raft_consensus.cc:2781] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:48.320853 32345 raft_consensus.cc:686] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:48.322158 32345 consensus_queue.cc:237] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:48.325448 27894 catalog_manager.cc:5484] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:48.340698 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:48.349654 32585 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:48.350807 32588 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:48.350617 32585 tablet.cc:1620] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:48.351142 32586 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:48.351274 32588 tablet.cc:1620] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:48.351562 32586 tablet.cc:1620] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:48.353340 32481 raft_consensus.cc:482] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:48.353506 32481 raft_consensus.cc:504] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.354396 32481 leader_election.cc:288] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:48.355057 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b263006614c46b4ac0648efb7a37a8f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:48.355255 30941 raft_consensus.cc:2454] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:48.355309 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b263006614c46b4ac0648efb7a37a8f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:48.355523 31072 raft_consensus.cc:2454] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:48.355973 27978 leader_election.cc:302] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:48.356694 32481 raft_consensus.cc:2781] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:48.356820 32481 raft_consensus.cc:482] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:48.357460 32481 raft_consensus.cc:3037] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.362998 32481 raft_consensus.cc:504] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.363564 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=eb88d0f4f9e3447da11754a4bb6a75d5] alter complete (version 2)
I20240503 14:13:48.364013 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b263006614c46b4ac0648efb7a37a8f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:48.364207 32481 leader_election.cc:288] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:48.364251 30941 raft_consensus.cc:3037] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.364513 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b263006614c46b4ac0648efb7a37a8f" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:48.364717 31072 raft_consensus.cc:3037] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.367585 30941 raft_consensus.cc:2454] T 6b263006614c46b4ac0648efb7a37a8f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:48.368683 27978 leader_election.cc:302] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:48.370025 32481 raft_consensus.cc:2781] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:48.370286 32481 raft_consensus.cc:686] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:48.370584 32481 consensus_queue.cc:237] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.373320 31072 raft_consensus.cc:2454] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:48.389623 27894 catalog_manager.cc:5484] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:48.474160 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:48.483232 32585 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:48.483626 32586 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:48.491107 32588 tablet.cc:1722] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:48.495327 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=eb88d0f4f9e3447da11754a4bb6a75d5] alter complete (version 3)
I20240503 14:13:48.533457 32325 consensus_queue.cc:1035] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.537685 32325 consensus_queue.cc:1035] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.598728 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:48.598956 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:48.601739 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 6b25f3678e0c433e948ac0bb7e4a618d
I20240503 14:13:48.602577 28282 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b25f3678e0c433e948ac0bb7e4a618d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:48 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:48.602598 30920 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b25f3678e0c433e948ac0bb7e4a618d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:48 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:48.603214 32597 tablet_replica.cc:330] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:48.603451 32597 raft_consensus.cc:2229] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:48.603564 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b25f3678e0c433e948ac0bb7e4a618d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:48 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:48.603844 32597 raft_consensus.cc:2258] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:48.604187 32598 tablet_replica.cc:330] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:48.604449 32598 raft_consensus.cc:2229] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:48.604701 32598 raft_consensus.cc:2258] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:48.607038 32598 ts_tablet_manager.cc:1875] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:48.613404 32598 ts_tablet_manager.cc:1888] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:48.613804 32596 tablet_replica.cc:330] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:48.614041 32596 raft_consensus.cc:2229] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:48.614280 32596 raft_consensus.cc:2258] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:48.614588 32598 log.cc:1196] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/6b25f3678e0c433e948ac0bb7e4a618d
I20240503 14:13:48.615121 32597 ts_tablet_manager.cc:1875] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:48.615278 32596 ts_tablet_manager.cc:1875] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:48.615584 32598 ts_tablet_manager.cc:1909] T 6b25f3678e0c433e948ac0bb7e4a618d P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:48.616622 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 6b25f3678e0c433e948ac0bb7e4a618d (table rebalancer_dynamic_table_new_name [id=eb88d0f4f9e3447da11754a4bb6a75d5]) successfully deleted
I20240503 14:13:48.621909 32596 ts_tablet_manager.cc:1888] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:48.621985 32597 ts_tablet_manager.cc:1888] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:48.622041 32596 log.cc:1196] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/6b25f3678e0c433e948ac0bb7e4a618d
I20240503 14:13:48.622128 32597 log.cc:1196] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/6b25f3678e0c433e948ac0bb7e4a618d
I20240503 14:13:48.622556 32596 ts_tablet_manager.cc:1909] T 6b25f3678e0c433e948ac0bb7e4a618d P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:48.622606 32597 ts_tablet_manager.cc:1909] T 6b25f3678e0c433e948ac0bb7e4a618d P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:48.623579 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 6b25f3678e0c433e948ac0bb7e4a618d (table rebalancer_dynamic_table_new_name [id=eb88d0f4f9e3447da11754a4bb6a75d5]) successfully deleted
I20240503 14:13:48.623893 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 6b25f3678e0c433e948ac0bb7e4a618d (table rebalancer_dynamic_table_new_name [id=eb88d0f4f9e3447da11754a4bb6a75d5]) successfully deleted
I20240503 14:13:48.702723 32345 consensus_queue.cc:1035] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.704603 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:48.708961 32345 consensus_queue.cc:1035] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.720384 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 005dc209529d46358e4a93c6c97e274b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552]), partition=
I20240503 14:13:48.720583 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 005dc209529d46358e4a93c6c97e274b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552]), partition=
I20240503 14:13:48.720769 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005dc209529d46358e4a93c6c97e274b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.720901 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005dc209529d46358e4a93c6c97e274b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.726899 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 005dc209529d46358e4a93c6c97e274b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552]), partition=
I20240503 14:13:48.727283 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005dc209529d46358e4a93c6c97e274b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.734447 32603 tablet_bootstrap.cc:492] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:48.736593 32605 tablet_bootstrap.cc:492] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:48.737005 32604 tablet_bootstrap.cc:492] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:48.738739 32604 tablet_bootstrap.cc:654] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.738898 32605 tablet_bootstrap.cc:654] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.738899 32603 tablet_bootstrap.cc:654] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.745887 32605 tablet_bootstrap.cc:492] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:48.746043 32605 ts_tablet_manager.cc:1367] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240503 14:13:48.746174 32604 tablet_bootstrap.cc:492] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:48.746337 32604 ts_tablet_manager.cc:1367] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20240503 14:13:48.746909 32605 raft_consensus.cc:348] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.747067 32604 raft_consensus.cc:348] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.747179 32605 raft_consensus.cc:374] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.747278 32604 raft_consensus.cc:374] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.747289 32605 raft_consensus.cc:729] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.747375 32604 raft_consensus.cc:729] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.747527 32605 consensus_queue.cc:260] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.747601 32604 consensus_queue.cc:260] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.748080 32605 ts_tablet_manager.cc:1398] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:48.748126 32604 ts_tablet_manager.cc:1398] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:48.758692 32603 tablet_bootstrap.cc:492] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:48.758826 32603 ts_tablet_manager.cc:1367] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20240503 14:13:48.759378 32603 raft_consensus.cc:348] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.759598 32603 raft_consensus.cc:374] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.759698 32603 raft_consensus.cc:729] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.760345 32603 consensus_queue.cc:260] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.760886 32603 ts_tablet_manager.cc:1398] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:48.887450 32553 raft_consensus.cc:482] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:48.887641 32553 raft_consensus.cc:504] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.888427 32553 leader_election.cc:288] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:48.891553 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dc209529d46358e4a93c6c97e274b" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:48.891872 31072 raft_consensus.cc:2454] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:48.892755 31138 leader_election.cc:302] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:48.892578 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dc209529d46358e4a93c6c97e274b" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:48.893186 30941 raft_consensus.cc:2454] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:48.893462 32553 raft_consensus.cc:2781] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:48.893553 32553 raft_consensus.cc:482] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:48.893640 32553 raft_consensus.cc:3037] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.897441 32553 raft_consensus.cc:504] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.898576 32553 leader_election.cc:288] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:48.899878 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dc209529d46358e4a93c6c97e274b" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:48.900115 31072 raft_consensus.cc:3037] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.900043 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dc209529d46358e4a93c6c97e274b" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:48.900887 30940 raft_consensus.cc:3037] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.902947 31072 raft_consensus.cc:2454] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:48.903417 30940 raft_consensus.cc:2454] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:48.903565 31138 leader_election.cc:302] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:48.903863 32553 raft_consensus.cc:2781] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:48.904080 32553 raft_consensus.cc:686] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:48.904615 32481 consensus_queue.cc:1035] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.905958 32553 consensus_queue.cc:237] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:48.910223 27900 catalog_manager.cc:5484] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:48.912055 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:48.915488 32481 consensus_queue.cc:1035] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.923424 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 583063f9427749b3aacc667ec68e55b5 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.923698 30920 tablet_service.cc:1467] Processing CreateTablet for tablet 133f5691d69b461f8e7502ecffdb8a89 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.923925 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583063f9427749b3aacc667ec68e55b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.924137 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133f5691d69b461f8e7502ecffdb8a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.924324 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 583063f9427749b3aacc667ec68e55b5 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.924738 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583063f9427749b3aacc667ec68e55b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.926666 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 583063f9427749b3aacc667ec68e55b5 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.927055 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583063f9427749b3aacc667ec68e55b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.928603 28154 tablet_service.cc:1467] Processing CreateTablet for tablet a23db2d9b3864035a08f7965bfb36574 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.928984 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a23db2d9b3864035a08f7965bfb36574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.930438 32604 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:48.932308 32604 tablet_bootstrap.cc:654] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.933578 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 133f5691d69b461f8e7502ecffdb8a89 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.933912 28023 tablet_service.cc:1467] Processing CreateTablet for tablet a23db2d9b3864035a08f7965bfb36574 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.934000 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133f5691d69b461f8e7502ecffdb8a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.934185 30921 tablet_service.cc:1467] Processing CreateTablet for tablet a23db2d9b3864035a08f7965bfb36574 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.934329 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a23db2d9b3864035a08f7965bfb36574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.934581 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a23db2d9b3864035a08f7965bfb36574. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.942620 32604 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:48.942808 32604 ts_tablet_manager.cc:1367] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 14:13:48.943992 32604 raft_consensus.cc:348] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.944425 32604 raft_consensus.cc:374] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.944571 32604 raft_consensus.cc:729] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.944928 32604 consensus_queue.cc:260] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.947134 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 133f5691d69b461f8e7502ecffdb8a89 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0672b9d4f2f94999b19c7e1b2a2c7b12]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:48.947372 32604 ts_tablet_manager.cc:1398] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:48.947556 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133f5691d69b461f8e7502ecffdb8a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.947890 32617 tablet_bootstrap.cc:492] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:48.951031 32617 tablet_bootstrap.cc:654] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.953321 32619 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:48.956125 32617 tablet_bootstrap.cc:492] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:48.956274 32617 ts_tablet_manager.cc:1367] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240503 14:13:48.957095 32617 raft_consensus.cc:348] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.957479 32617 raft_consensus.cc:374] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.957672 32619 tablet_bootstrap.cc:654] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.958432 32617 raft_consensus.cc:729] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.958987 32617 consensus_queue.cc:260] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.960027 32604 tablet_bootstrap.cc:492] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:48.962698 32604 tablet_bootstrap.cc:654] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.964569 32616 tablet_bootstrap.cc:492] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:48.967497 32616 tablet_bootstrap.cc:654] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.969753 32617 ts_tablet_manager.cc:1398] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 14:13:48.970420 32619 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:48.970674 32619 ts_tablet_manager.cc:1367] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.004s
I20240503 14:13:48.971869 32619 raft_consensus.cc:348] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.972252 32619 raft_consensus.cc:374] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.972484 32619 raft_consensus.cc:729] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.972703 32605 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:48.972908 32619 consensus_queue.cc:260] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.974676 32605 tablet_bootstrap.cc:654] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.975471 32617 tablet_bootstrap.cc:492] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:48.976311 32619 ts_tablet_manager.cc:1398] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:48.976742 32619 tablet_bootstrap.cc:492] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:48.977766 32617 tablet_bootstrap.cc:654] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.978904 32619 tablet_bootstrap.cc:654] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.982332 32621 tablet_bootstrap.cc:492] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:48.985672 32604 tablet_bootstrap.cc:492] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:48.985958 32604 ts_tablet_manager.cc:1367] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.001s
I20240503 14:13:48.987280 32604 raft_consensus.cc:348] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.987628 32604 raft_consensus.cc:374] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.987861 32604 raft_consensus.cc:729] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.988350 32605 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:48.988324 32604 consensus_queue.cc:260] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:48.988585 32605 ts_tablet_manager.cc:1367] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240503 14:13:48.989648 32605 raft_consensus.cc:348] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.989888 32605 raft_consensus.cc:374] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:48.989971 32605 raft_consensus.cc:729] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:48.992581 32604 ts_tablet_manager.cc:1398] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:48.990276 32605 consensus_queue.cc:260] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:48.997586 32605 ts_tablet_manager.cc:1398] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:48.998184 32621 tablet_bootstrap.cc:654] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:48.998728 32616 tablet_bootstrap.cc:492] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:48.998888 32616 ts_tablet_manager.cc:1367] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.022s
I20240503 14:13:49.000247 32616 raft_consensus.cc:348] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.000504 32616 raft_consensus.cc:374] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.000694 32616 raft_consensus.cc:729] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.001109 32616 consensus_queue.cc:260] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.001854 32616 ts_tablet_manager.cc:1398] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:49.003587 32617 tablet_bootstrap.cc:492] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:49.003587 32619 tablet_bootstrap.cc:492] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:49.003813 32619 ts_tablet_manager.cc:1367] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
I20240503 14:13:49.003947 32617 ts_tablet_manager.cc:1367] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.006s
I20240503 14:13:49.005157 32617 raft_consensus.cc:348] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.005334 32619 raft_consensus.cc:348] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.005499 32617 raft_consensus.cc:374] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.005604 32619 raft_consensus.cc:374] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.005728 32619 raft_consensus.cc:729] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.006059 32619 consensus_queue.cc:260] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.006595 32619 ts_tablet_manager.cc:1398] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:49.007452 32621 tablet_bootstrap.cc:492] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:49.007678 32621 ts_tablet_manager.cc:1367] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.007s
I20240503 14:13:49.005751 32617 raft_consensus.cc:729] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.008395 32617 consensus_queue.cc:260] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.008877 32621 raft_consensus.cc:348] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.009135 32621 raft_consensus.cc:374] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.009279 32617 ts_tablet_manager.cc:1398] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:49.009264 32621 raft_consensus.cc:729] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.009558 32621 consensus_queue.cc:260] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.010655 32621 ts_tablet_manager.cc:1398] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:49.022368 27900 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:49.027943 31072 raft_consensus.cc:1261] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:49.027961 30940 raft_consensus.cc:1261] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:49.028594 32611 consensus_queue.cc:1035] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.028856 32610 consensus_queue.cc:1035] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.031406 32580 raft_consensus.cc:482] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:49.031572 32580 raft_consensus.cc:504] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.032586 32580 leader_election.cc:288] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:49.033322 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583063f9427749b3aacc667ec68e55b5" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:49.033736 28432 raft_consensus.cc:2454] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:49.034368 32594 mvcc.cc:205] Tried to move back new op lower bound from 7023598096492736512 to 7023598096001540096. Current Snapshot: MvccSnapshot[applied={T|T < 7023598096492736512}]
I20240503 14:13:49.035547 28240 leader_election.cc:302] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:49.037252 32580 raft_consensus.cc:2781] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:49.038144 32517 raft_consensus.cc:482] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:49.038430 32517 raft_consensus.cc:504] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.038985 32633 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:49.039335 32517 leader_election.cc:288] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:49.039767 32633 tablet.cc:1620] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:49.040463 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23db2d9b3864035a08f7965bfb36574" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:49.040585 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23db2d9b3864035a08f7965bfb36574" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:49.040810 28174 raft_consensus.cc:2454] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:49.040764 28040 raft_consensus.cc:2454] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:49.041335 30877 leader_election.cc:302] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:49.043777 32517 raft_consensus.cc:2781] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:49.043913 32517 raft_consensus.cc:482] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:49.044070 32517 raft_consensus.cc:3037] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.045433 32632 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:49.045737 32580 raft_consensus.cc:482] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:49.045908 32580 raft_consensus.cc:3037] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.045861 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583063f9427749b3aacc667ec68e55b5" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:49.046062 28040 raft_consensus.cc:2454] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:49.046613 32632 tablet.cc:1620] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:49.051059 27894 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 005dc209529d46358e4a93c6c97e274b (table rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552]). Expected version 1 got 0
I20240503 14:13:49.048872 32517 raft_consensus.cc:504] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.052935 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23db2d9b3864035a08f7965bfb36574" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:49.053037 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23db2d9b3864035a08f7965bfb36574" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:49.053150 28040 raft_consensus.cc:3037] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.053275 28174 raft_consensus.cc:3037] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.056401 28174 raft_consensus.cc:2454] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:49.056815 30875 leader_election.cc:302] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:49.057353 32190 raft_consensus.cc:2781] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:49.058430 32580 raft_consensus.cc:504] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.059684 32190 raft_consensus.cc:686] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:49.060051 32190 consensus_queue.cc:237] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.061566 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583063f9427749b3aacc667ec68e55b5" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:49.061877 28432 raft_consensus.cc:3037] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.062350 32580 leader_election.cc:288] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:49.063794 32634 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:49.064584 32634 tablet.cc:1620] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:49.064741 27900 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 005dc209529d46358e4a93c6c97e274b (table rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552]). Expected version 1 got 0
I20240503 14:13:49.064744 28432 raft_consensus.cc:2454] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:49.065232 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583063f9427749b3aacc667ec68e55b5" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:49.065486 28042 raft_consensus.cc:3037] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.066356 32611 raft_consensus.cc:482] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:49.066605 32611 raft_consensus.cc:504] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.067670 32611 leader_election.cc:288] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:49.067855 27894 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet 005dc209529d46358e4a93c6c97e274b (table rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552]). Expected version 1 got 0
I20240503 14:13:49.068367 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133f5691d69b461f8e7502ecffdb8a89" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:49.068547 28042 raft_consensus.cc:2454] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:49.068667 30940 raft_consensus.cc:2454] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:49.069178 31138 leader_election.cc:302] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:49.069684 32611 raft_consensus.cc:2781] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:49.069811 32611 raft_consensus.cc:482] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:49.069975 32611 raft_consensus.cc:3037] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.070413 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133f5691d69b461f8e7502ecffdb8a89" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:49.070708 28304 raft_consensus.cc:2454] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:49.073987 32611 raft_consensus.cc:504] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.075244 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133f5691d69b461f8e7502ecffdb8a89" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:49.075471 30939 raft_consensus.cc:3037] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.076021 32611 leader_election.cc:288] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:49.077435 28040 raft_consensus.cc:2454] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:49.079948 28240 leader_election.cc:302] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:49.080756 32376 raft_consensus.cc:2781] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:49.080947 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "133f5691d69b461f8e7502ecffdb8a89" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:49.080863 27899 catalog_manager.cc:5484] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:49.081317 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552] alter complete (version 1)
I20240503 14:13:49.081357 28302 raft_consensus.cc:3037] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.084697 28302 raft_consensus.cc:2454] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:49.088498 31139 leader_election.cc:302] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:49.089071 32611 raft_consensus.cc:2781] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:49.089504 32611 raft_consensus.cc:686] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:49.089888 32611 consensus_queue.cc:237] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.079820 32517 leader_election.cc:288] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:49.084153 30939 raft_consensus.cc:2454] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:49.091161 32376 raft_consensus.cc:686] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:49.091527 32376 consensus_queue.cc:237] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:49.096583 27899 catalog_manager.cc:5484] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:49.102900 27899 catalog_manager.cc:5484] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:49.184759 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:49.192250 32632 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:49.192836 32632 tablet.cc:1620] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:49.193311 32634 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:49.193773 32634 tablet.cc:1620] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:49.194687 32633 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:49.195147 32633 tablet.cc:1620] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:49.202611 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=32a25dcd5cde4ae6b9bc1b252b266552] alter complete (version 2)
I20240503 14:13:49.305490 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:49.312965 32632 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:49.313830 32634 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:49.314038 32633 tablet.cc:1722] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:49.322420 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=32a25dcd5cde4ae6b9bc1b252b266552] alter complete (version 3)
I20240503 14:13:49.424506 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:49.424755 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:49.427770 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 005dc209529d46358e4a93c6c97e274b
I20240503 14:13:49.428494 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 005dc209529d46358e4a93c6c97e274b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:49 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:49.428606 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 005dc209529d46358e4a93c6c97e274b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:49 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:49.428697 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 005dc209529d46358e4a93c6c97e274b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:49 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:49.429143 32642 tablet_replica.cc:330] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:49.429438 32642 raft_consensus.cc:2229] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:49.429680 32642 raft_consensus.cc:2258] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:49.430595 32642 ts_tablet_manager.cc:1875] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:49.439227 32642 ts_tablet_manager.cc:1888] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:49.439244 32644 tablet_replica.cc:330] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:49.439422 32642 log.cc:1196] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/005dc209529d46358e4a93c6c97e274b
I20240503 14:13:49.439494 32644 raft_consensus.cc:2229] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:49.439930 32644 raft_consensus.cc:2258] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:49.440021 32642 ts_tablet_manager.cc:1909] T 005dc209529d46358e4a93c6c97e274b P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:49.440373 32643 tablet_replica.cc:330] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:49.440657 32643 raft_consensus.cc:2229] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:49.441172 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 005dc209529d46358e4a93c6c97e274b (table rebalancer_dynamic_table_new_name [id=32a25dcd5cde4ae6b9bc1b252b266552]) successfully deleted
I20240503 14:13:49.441475 32643 raft_consensus.cc:2258] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:49.442425 32644 ts_tablet_manager.cc:1875] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:49.442577 32643 ts_tablet_manager.cc:1875] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:49.446311 32636 consensus_queue.cc:1035] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.449440 32643 ts_tablet_manager.cc:1888] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:49.449606 32643 log.cc:1196] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/005dc209529d46358e4a93c6c97e274b
I20240503 14:13:49.450199 32643 ts_tablet_manager.cc:1909] T 005dc209529d46358e4a93c6c97e274b P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:49.450215 32644 ts_tablet_manager.cc:1888] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:49.450578 32644 log.cc:1196] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/005dc209529d46358e4a93c6c97e274b
I20240503 14:13:49.451308 32644 ts_tablet_manager.cc:1909] T 005dc209529d46358e4a93c6c97e274b P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:49.451372 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 005dc209529d46358e4a93c6c97e274b (table rebalancer_dynamic_table_new_name [id=32a25dcd5cde4ae6b9bc1b252b266552]) successfully deleted
I20240503 14:13:49.452603 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 005dc209529d46358e4a93c6c97e274b (table rebalancer_dynamic_table_new_name [id=32a25dcd5cde4ae6b9bc1b252b266552]) successfully deleted
I20240503 14:13:49.461678 32636 consensus_queue.cc:1035] T a23db2d9b3864035a08f7965bfb36574 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.485709 32376 consensus_queue.cc:1035] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.504280 32376 consensus_queue.cc:1035] T 583063f9427749b3aacc667ec68e55b5 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.530684 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:49.539362 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8bf831af114c97aa3e911cc17707c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d]), partition=
I20240503 14:13:49.539456 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8bf831af114c97aa3e911cc17707c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d]), partition=
I20240503 14:13:49.539779 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8bf831af114c97aa3e911cc17707c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.539819 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8bf831af114c97aa3e911cc17707c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.546072 32654 tablet_bootstrap.cc:492] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:49.546053 32653 tablet_bootstrap.cc:492] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:49.546252 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 5b8bf831af114c97aa3e911cc17707c3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d]), partition=
I20240503 14:13:49.546749 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b8bf831af114c97aa3e911cc17707c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.548317 32654 tablet_bootstrap.cc:654] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.548810 32653 tablet_bootstrap.cc:654] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.558151 32653 tablet_bootstrap.cc:492] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:49.558586 32653 ts_tablet_manager.cc:1367] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240503 14:13:49.560010 32653 raft_consensus.cc:348] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.560348 32653 raft_consensus.cc:374] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.560575 32653 raft_consensus.cc:729] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.560889 32653 consensus_queue.cc:260] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.561586 32653 ts_tablet_manager.cc:1398] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:49.564330 32657 tablet_bootstrap.cc:492] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:49.566807 32657 tablet_bootstrap.cc:654] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.569545 32654 tablet_bootstrap.cc:492] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:49.569774 32654 ts_tablet_manager.cc:1367] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.000s
I20240503 14:13:49.569882 32657 tablet_bootstrap.cc:492] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:49.570202 32657 ts_tablet_manager.cc:1367] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:13:49.570823 32654 raft_consensus.cc:348] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.571132 32654 raft_consensus.cc:374] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.571309 32654 raft_consensus.cc:729] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.571309 32657 raft_consensus.cc:348] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.571563 32657 raft_consensus.cc:374] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.571659 32654 consensus_queue.cc:260] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.571735 32657 raft_consensus.cc:729] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.572077 32657 consensus_queue.cc:260] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.572283 32654 ts_tablet_manager.cc:1398] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:49.576895 32657 ts_tablet_manager.cc:1398] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:49.589396 32611 consensus_queue.cc:1035] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.594226 32553 consensus_queue.cc:1035] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.619282 32638 raft_consensus.cc:482] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:49.619549 32638 raft_consensus.cc:504] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.620692 32638 leader_election.cc:288] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:49.621098 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bf831af114c97aa3e911cc17707c3" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:49.621380 30808 raft_consensus.cc:2454] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:49.621848 28237 leader_election.cc:302] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:49.622123 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bf831af114c97aa3e911cc17707c3" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:49.622483 31203 raft_consensus.cc:2454] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:49.622515 32638 raft_consensus.cc:2781] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:49.622671 32638 raft_consensus.cc:482] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:49.622819 32638 raft_consensus.cc:3037] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.626057 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:49.626081 32638 raft_consensus.cc:504] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.629215 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bf831af114c97aa3e911cc17707c3" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:49.629462 30808 raft_consensus.cc:3037] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.630318 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b8bf831af114c97aa3e911cc17707c3" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:49.630709 31203 raft_consensus.cc:3037] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.632961 30808 raft_consensus.cc:2454] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:49.634572 28237 leader_election.cc:302] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:49.636727 32376 raft_consensus.cc:2781] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:49.639122 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 9088c0106bf249cab5b35c1171a7e7a0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.639276 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 9088c0106bf249cab5b35c1171a7e7a0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.639668 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9088c0106bf249cab5b35c1171a7e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.639860 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9088c0106bf249cab5b35c1171a7e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.640272 30920 tablet_service.cc:1467] Processing CreateTablet for tablet b7b96e9a7f314f84a38fee177ade3da2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.640792 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b96e9a7f314f84a38fee177ade3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.641860 31203 raft_consensus.cc:2454] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:49.643008 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 9088c0106bf249cab5b35c1171a7e7a0 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.643668 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9088c0106bf249cab5b35c1171a7e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.629817 32638 leader_election.cc:288] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:49.649446 28154 tablet_service.cc:1467] Processing CreateTablet for tablet b7b96e9a7f314f84a38fee177ade3da2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.650031 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b96e9a7f314f84a38fee177ade3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.655043 31182 tablet_service.cc:1467] Processing CreateTablet for tablet b7b96e9a7f314f84a38fee177ade3da2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.655675 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b96e9a7f314f84a38fee177ade3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.659183 30919 tablet_service.cc:1467] Processing CreateTablet for tablet 029be0389e244435bb08db2361729ad2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.659653 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029be0389e244435bb08db2361729ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.662343 32376 raft_consensus.cc:686] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:49.662765 32376 consensus_queue.cc:237] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.669414 28414 tablet_service.cc:1467] Processing CreateTablet for tablet 029be0389e244435bb08db2361729ad2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.669975 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029be0389e244435bb08db2361729ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.670094 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 029be0389e244435bb08db2361729ad2 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:49.670589 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029be0389e244435bb08db2361729ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.677500 32663 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:49.679973 32663 tablet_bootstrap.cc:654] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.686370 32665 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:49.687160 32657 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:49.689114 32657 tablet_bootstrap.cc:654] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.692124 32665 tablet_bootstrap.cc:654] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.702569 32665 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:49.702741 32665 ts_tablet_manager.cc:1367] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240503 14:13:49.703712 32665 raft_consensus.cc:348] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.703979 32665 raft_consensus.cc:374] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.704073 32665 raft_consensus.cc:729] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.704280 32665 consensus_queue.cc:260] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.704980 32665 ts_tablet_manager.cc:1398] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:49.710287 32667 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:49.712728 32667 tablet_bootstrap.cc:654] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.713444 32668 tablet_bootstrap.cc:492] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
W20240503 14:13:49.717370 28240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 81689.3us
I20240503 14:13:49.718489 32657 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:49.718827 32657 ts_tablet_manager.cc:1367] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20240503 14:13:49.719974 32657 raft_consensus.cc:348] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.720508 32657 raft_consensus.cc:374] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.720685 32657 raft_consensus.cc:729] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.721105 32657 consensus_queue.cc:260] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.723888 32657 ts_tablet_manager.cc:1398] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:49.724336 32657 tablet_bootstrap.cc:492] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:49.726604 32668 tablet_bootstrap.cc:654] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.740141 32657 tablet_bootstrap.cc:654] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.748342 32668 tablet_bootstrap.cc:492] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:49.748560 32668 ts_tablet_manager.cc:1367] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.003s
I20240503 14:13:49.748723 32657 tablet_bootstrap.cc:492] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:49.748904 32657 ts_tablet_manager.cc:1367] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240503 14:13:49.749707 32668 raft_consensus.cc:348] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.750085 32668 raft_consensus.cc:374] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.750324 32668 raft_consensus.cc:729] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.750672 32668 consensus_queue.cc:260] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.751578 32668 ts_tablet_manager.cc:1398] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:49.751700 32667 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:49.751893 32663 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:49.752370 32667 ts_tablet_manager.cc:1367] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20240503 14:13:49.754657 32663 ts_tablet_manager.cc:1367] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.075s user 0.022s sys 0.026s
I20240503 14:13:49.754824 32667 raft_consensus.cc:348] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.755223 32667 raft_consensus.cc:374] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.755553 27899 catalog_manager.cc:5484] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:49.755834 32663 raft_consensus.cc:348] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.756052 32663 raft_consensus.cc:374] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.756208 32663 raft_consensus.cc:729] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.756471 32663 consensus_queue.cc:260] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.757133 32663 ts_tablet_manager.cc:1398] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:49.757417 32667 raft_consensus.cc:729] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.757606 32663 tablet_bootstrap.cc:492] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:49.757804 32667 consensus_queue.cc:260] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.758549 32667 ts_tablet_manager.cc:1398] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:49.758915 32657 raft_consensus.cc:348] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.759191 32657 raft_consensus.cc:374] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.759388 32657 raft_consensus.cc:729] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.761152 32635 raft_consensus.cc:482] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:49.761353 32635 raft_consensus.cc:504] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.762367 32635 leader_election.cc:288] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:49.762856 32667 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:49.763079 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9088c0106bf249cab5b35c1171a7e7a0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:49.763309 28434 raft_consensus.cc:2454] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:49.763397 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9088c0106bf249cab5b35c1171a7e7a0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:49.763653 31203 raft_consensus.cc:2454] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:49.759663 32657 consensus_queue.cc:260] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.764282 30875 leader_election.cc:302] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:49.764715 32635 raft_consensus.cc:2781] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:49.764894 32635 raft_consensus.cc:482] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:49.765038 32635 raft_consensus.cc:3037] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.768846 32635 raft_consensus.cc:504] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.770221 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9088c0106bf249cab5b35c1171a7e7a0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:49.770561 28434 raft_consensus.cc:3037] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.771098 32667 tablet_bootstrap.cc:654] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.771597 32657 ts_tablet_manager.cc:1398] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 14:13:49.771957 32635 leader_election.cc:288] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:49.772341 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9088c0106bf249cab5b35c1171a7e7a0" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:49.772567 31203 raft_consensus.cc:3037] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.776069 31203 raft_consensus.cc:2454] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:49.776341 28434 raft_consensus.cc:2454] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:49.776494 30875 leader_election.cc:302] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:49.776893 32636 raft_consensus.cc:2781] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:49.777069 32636 raft_consensus.cc:686] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:49.777479 32636 consensus_queue.cc:237] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.780230 32667 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:49.780503 32667 ts_tablet_manager.cc:1367] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240503 14:13:49.785537 32667 raft_consensus.cc:348] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.785746 32663 tablet_bootstrap.cc:654] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.785949 32667 raft_consensus.cc:374] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.786288 32667 raft_consensus.cc:729] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.786764 32667 consensus_queue.cc:260] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.789502 32667 ts_tablet_manager.cc:1398] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 14:13:49.792057 32663 tablet_bootstrap.cc:492] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:49.792322 32663 ts_tablet_manager.cc:1367] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20240503 14:13:49.793794 32663 raft_consensus.cc:348] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.794147 32663 raft_consensus.cc:374] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.794296 32663 raft_consensus.cc:729] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.794273 27898 catalog_manager.cc:5484] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:49.794785 32663 consensus_queue.cc:260] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.795562 32663 ts_tablet_manager.cc:1398] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:13:49.795980 32663 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:49.797747 32663 tablet_bootstrap.cc:654] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:49.800021 32663 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:49.800148 32663 ts_tablet_manager.cc:1367] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:49.800807 32663 raft_consensus.cc:348] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.800962 32663 raft_consensus.cc:374] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:49.801069 32663 raft_consensus.cc:729] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:49.801281 32663 consensus_queue.cc:260] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.801744 32663 ts_tablet_manager.cc:1398] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:49.897684 32553 raft_consensus.cc:482] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:49.897866 32553 raft_consensus.cc:504] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.898547 32553 leader_election.cc:288] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:49.899021 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b96e9a7f314f84a38fee177ade3da2" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:49.899281 30939 raft_consensus.cc:2454] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:49.899425 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b96e9a7f314f84a38fee177ade3da2" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:49.899636 28174 raft_consensus.cc:2454] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:49.899730 31138 leader_election.cc:302] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:49.900012 32553 raft_consensus.cc:2781] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:49.900142 32553 raft_consensus.cc:482] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:49.900229 32553 raft_consensus.cc:3037] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.902174 32553 raft_consensus.cc:504] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.902724 32553 leader_election.cc:288] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:49.903120 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b96e9a7f314f84a38fee177ade3da2" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:49.903301 30939 raft_consensus.cc:3037] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.903240 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b96e9a7f314f84a38fee177ade3da2" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:49.903527 28174 raft_consensus.cc:3037] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.905586 30939 raft_consensus.cc:2454] T b7b96e9a7f314f84a38fee177ade3da2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:49.905977 31138 leader_election.cc:302] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:49.906085 28174 raft_consensus.cc:2454] T b7b96e9a7f314f84a38fee177ade3da2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:49.906213 32610 raft_consensus.cc:2781] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:49.906416 32610 raft_consensus.cc:686] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:49.906736 32610 consensus_queue.cc:237] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:49.910507 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:49.911949 27898 catalog_manager.cc:5484] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:49.919279 31203 raft_consensus.cc:1261] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:49.920218 30808 raft_consensus.cc:1261] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:49.920305 32376 consensus_queue.cc:1035] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.920995 32376 consensus_queue.cc:1035] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:49.921140 32230 raft_consensus.cc:482] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:49.921322 32230 raft_consensus.cc:504] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.922165 32230 leader_election.cc:288] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:49.922878 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029be0389e244435bb08db2361729ad2" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:49.923095 28040 raft_consensus.cc:2454] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:49.923827 28370 leader_election.cc:302] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:49.924269 32230 raft_consensus.cc:2781] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:49.924378 32230 raft_consensus.cc:482] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:49.924499 32230 raft_consensus.cc:3037] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.925894 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029be0389e244435bb08db2361729ad2" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:49.926100 30939 raft_consensus.cc:2454] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:49.929442 27899 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 5b8bf831af114c97aa3e911cc17707c3 (table rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d]). Expected version 1 got 0
I20240503 14:13:49.931612 32230 raft_consensus.cc:504] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.933898 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029be0389e244435bb08db2361729ad2" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:49.934139 30939 raft_consensus.cc:3037] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.936606 27899 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 5b8bf831af114c97aa3e911cc17707c3 (table rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d]). Expected version 1 got 0
I20240503 14:13:49.936879 30939 raft_consensus.cc:2454] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:49.937415 28369 leader_election.cc:302] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:49.937709 32595 raft_consensus.cc:2781] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:49.938416 32685 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:49.939215 32685 tablet.cc:1620] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:49.941466 27899 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 5b8bf831af114c97aa3e911cc17707c3 (table rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d]). Expected version 1 got 0
I20240503 14:13:49.946328 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029be0389e244435bb08db2361729ad2" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:49.946530 28040 raft_consensus.cc:3037] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:49.945989 32682 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:49.949700 32682 tablet.cc:1620] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:49.949793 28040 raft_consensus.cc:2454] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:49.951460 32230 leader_election.cc:288] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:49.954370 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d] alter complete (version 1)
I20240503 14:13:49.964524 32595 raft_consensus.cc:686] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:49.964920 32595 consensus_queue.cc:237] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:49.967221 32683 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:49.967873 32683 tablet.cc:1620] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:49.972348 27898 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 5b8bf831af114c97aa3e911cc17707c3 (table rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d]). Expected version 1 got 0
I20240503 14:13:49.977887 27898 catalog_manager.cc:5484] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:50.061177 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:50.070160 32685 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:50.070658 32685 tablet.cc:1620] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:50.071962 32683 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:50.072551 32683 tablet.cc:1620] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:50.073102 32682 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:50.078298 32682 tablet.cc:1620] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:50.085222 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=e955953b402646d08d8f4802ae31c58d] alter complete (version 2)
I20240503 14:13:50.190742 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:50.197991 32682 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:50.198402 32683 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:50.203917 32685 tablet.cc:1722] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:50.208138 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=e955953b402646d08d8f4802ae31c58d] alter complete (version 3)
I20240503 14:13:50.311475 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:50.311713 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:50.314882 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 5b8bf831af114c97aa3e911cc17707c3
I20240503 14:13:50.315665 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8bf831af114c97aa3e911cc17707c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:50 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:50.315706 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8bf831af114c97aa3e911cc17707c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:50 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:50.315794 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b8bf831af114c97aa3e911cc17707c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:50 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:50.316953 32692 tablet_replica.cc:330] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:50.317339 32692 raft_consensus.cc:2229] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:50.317672 32692 raft_consensus.cc:2258] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:50.318292 32691 tablet_replica.cc:330] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:50.318607 32691 raft_consensus.cc:2229] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:50.318826 32691 raft_consensus.cc:2258] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:50.319198 32692 ts_tablet_manager.cc:1875] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:50.319315 32690 tablet_replica.cc:330] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:50.319599 32690 raft_consensus.cc:2229] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:50.319931 32691 ts_tablet_manager.cc:1875] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:50.321516 32690 raft_consensus.cc:2258] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:50.322021 32635 consensus_queue.cc:1035] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.322533 32690 ts_tablet_manager.cc:1875] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:50.322908 32553 consensus_queue.cc:1035] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.328799 32691 ts_tablet_manager.cc:1888] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:50.329001 32691 log.cc:1196] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/5b8bf831af114c97aa3e911cc17707c3
I20240503 14:13:50.329572 32691 ts_tablet_manager.cc:1909] T 5b8bf831af114c97aa3e911cc17707c3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:50.329629 32690 ts_tablet_manager.cc:1888] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:50.329857 32690 log.cc:1196] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/5b8bf831af114c97aa3e911cc17707c3
I20240503 14:13:50.330288 32692 ts_tablet_manager.cc:1888] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:50.330493 32690 ts_tablet_manager.cc:1909] T 5b8bf831af114c97aa3e911cc17707c3 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:50.330475 32692 log.cc:1196] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/5b8bf831af114c97aa3e911cc17707c3
I20240503 14:13:50.331431 32692 ts_tablet_manager.cc:1909] T 5b8bf831af114c97aa3e911cc17707c3 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:50.332434 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 5b8bf831af114c97aa3e911cc17707c3 (table rebalancer_dynamic_table_new_name [id=e955953b402646d08d8f4802ae31c58d]) successfully deleted
I20240503 14:13:50.335271 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 5b8bf831af114c97aa3e911cc17707c3 (table rebalancer_dynamic_table_new_name [id=e955953b402646d08d8f4802ae31c58d]) successfully deleted
I20240503 14:13:50.337532 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 5b8bf831af114c97aa3e911cc17707c3 (table rebalancer_dynamic_table_new_name [id=e955953b402646d08d8f4802ae31c58d]) successfully deleted
I20240503 14:13:50.344126 32635 consensus_queue.cc:1035] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.349854 32610 consensus_queue.cc:1035] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.369958 32686 consensus_queue.cc:1035] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.398063 32230 consensus_queue.cc:1035] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.417409 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:50.429484 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 02180bd9a3704174a48ab282eecb619b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005]), partition=
I20240503 14:13:50.429709 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 02180bd9a3704174a48ab282eecb619b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005]), partition=
I20240503 14:13:50.429981 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02180bd9a3704174a48ab282eecb619b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.430076 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02180bd9a3704174a48ab282eecb619b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.430055 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 02180bd9a3704174a48ab282eecb619b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005]), partition=
I20240503 14:13:50.430395 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02180bd9a3704174a48ab282eecb619b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.434818 32703 tablet_bootstrap.cc:492] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:50.441339 32703 tablet_bootstrap.cc:654] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.446468 32706 tablet_bootstrap.cc:492] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:50.448706 32703 tablet_bootstrap.cc:492] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:50.448943 32706 tablet_bootstrap.cc:654] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.449023 32703 ts_tablet_manager.cc:1367] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20240503 14:13:50.449996 32703 raft_consensus.cc:348] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.452939 32706 tablet_bootstrap.cc:492] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:50.453143 32706 ts_tablet_manager.cc:1367] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240503 14:13:50.454313 32703 raft_consensus.cc:374] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.454439 32703 raft_consensus.cc:729] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.454335 32706 raft_consensus.cc:348] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.454581 32706 raft_consensus.cc:374] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.454687 32706 raft_consensus.cc:729] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.454720 32703 consensus_queue.cc:260] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.454948 32706 consensus_queue.cc:260] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.455474 32703 ts_tablet_manager.cc:1398] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:13:50.455909 32704 tablet_bootstrap.cc:492] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:50.458032 32706 ts_tablet_manager.cc:1398] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:50.459492 32704 tablet_bootstrap.cc:654] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.465453 32704 tablet_bootstrap.cc:492] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:50.465715 32704 ts_tablet_manager.cc:1367] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.002s
I20240503 14:13:50.468591 32704 raft_consensus.cc:348] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.468854 32704 raft_consensus.cc:374] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.468957 32704 raft_consensus.cc:729] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.469332 32704 consensus_queue.cc:260] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.470109 32704 ts_tablet_manager.cc:1398] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:50.516096 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:50.525193 28022 tablet_service.cc:1467] Processing CreateTablet for tablet 38cf3aab9e1b4f28842ac16d02be2562 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.525678 28022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cf3aab9e1b4f28842ac16d02be2562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.526084 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 81973d2082ab4bacb61722e51c146f30 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.526618 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81973d2082ab4bacb61722e51c146f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.528352 28153 tablet_service.cc:1467] Processing CreateTablet for tablet 81973d2082ab4bacb61722e51c146f30 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.528741 28153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81973d2082ab4bacb61722e51c146f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.528900 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 81973d2082ab4bacb61722e51c146f30 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.529349 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81973d2082ab4bacb61722e51c146f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.534601 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 38cf3aab9e1b4f28842ac16d02be2562 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.534967 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cf3aab9e1b4f28842ac16d02be2562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.535845 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd90464f4b84baabd90a44a122e28b1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.536294 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd90464f4b84baabd90a44a122e28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.537173 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd90464f4b84baabd90a44a122e28b1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.537535 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd90464f4b84baabd90a44a122e28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.538841 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 38cf3aab9e1b4f28842ac16d02be2562 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.539418 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cf3aab9e1b4f28842ac16d02be2562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.543097 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 8dd90464f4b84baabd90a44a122e28b1 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:50.543552 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd90464f4b84baabd90a44a122e28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:50.543715 32710 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:50.545313 32710 tablet_bootstrap.cc:654] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.562747 32711 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:50.564299 32711 tablet_bootstrap.cc:654] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.564658 32710 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:50.564855 32710 ts_tablet_manager.cc:1367] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20240503 14:13:50.565565 32704 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:50.566052 32710 raft_consensus.cc:348] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.566496 32710 raft_consensus.cc:374] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.566669 32710 raft_consensus.cc:729] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.566987 32710 consensus_queue.cc:260] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.567214 32713 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:50.567365 32704 tablet_bootstrap.cc:654] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.567610 32710 ts_tablet_manager.cc:1398] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:50.567900 32710 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:50.570011 32710 tablet_bootstrap.cc:654] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.577118 32713 tablet_bootstrap.cc:654] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.577734 32704 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:50.577980 32704 ts_tablet_manager.cc:1367] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:13:50.578799 32704 raft_consensus.cc:348] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.579687 32711 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:50.579893 32711 ts_tablet_manager.cc:1367] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240503 14:13:50.580729 32711 raft_consensus.cc:348] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.581043 32704 raft_consensus.cc:374] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.581179 32704 raft_consensus.cc:729] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.581497 32704 consensus_queue.cc:260] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.582159 32704 ts_tablet_manager.cc:1398] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:50.582319 32711 raft_consensus.cc:374] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.582474 32711 raft_consensus.cc:729] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.583236 32711 consensus_queue.cc:260] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.583498 32710 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:50.583712 32710 ts_tablet_manager.cc:1367] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20240503 14:13:50.584035 32709 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:50.584532 32710 raft_consensus.cc:348] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.584813 32710 raft_consensus.cc:374] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.584911 32710 raft_consensus.cc:729] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.585271 32711 ts_tablet_manager.cc:1398] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:50.585225 32710 consensus_queue.cc:260] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.585942 32710 ts_tablet_manager.cc:1398] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:13:50.586293 32711 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:50.586890 32713 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:50.587057 32713 ts_tablet_manager.cc:1367] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20240503 14:13:50.587808 32713 raft_consensus.cc:348] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.588034 32713 raft_consensus.cc:374] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.588135 32713 raft_consensus.cc:729] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.588359 32713 consensus_queue.cc:260] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.588840 32709 tablet_bootstrap.cc:654] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.589104 32713 ts_tablet_manager.cc:1398] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:50.590288 32711 tablet_bootstrap.cc:654] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.599946 32711 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:50.600124 32711 ts_tablet_manager.cc:1367] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20240503 14:13:50.600901 32711 raft_consensus.cc:348] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.601145 32711 raft_consensus.cc:374] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.601227 32711 raft_consensus.cc:729] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.601459 32711 consensus_queue.cc:260] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.601930 32711 ts_tablet_manager.cc:1398] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:50.602205 32711 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:50.603889 32711 tablet_bootstrap.cc:654] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.605525 32709 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:50.605674 32709 ts_tablet_manager.cc:1367] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.004s
I20240503 14:13:50.606402 32709 raft_consensus.cc:348] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.606606 32709 raft_consensus.cc:374] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.606696 32709 raft_consensus.cc:729] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.606906 32709 consensus_queue.cc:260] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.607084 32711 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:50.607237 32711 ts_tablet_manager.cc:1367] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:13:50.607412 32709 ts_tablet_manager.cc:1398] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:50.607697 32709 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:50.607901 32711 raft_consensus.cc:348] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.608119 32711 raft_consensus.cc:374] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.608237 32711 raft_consensus.cc:729] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.608508 32711 consensus_queue.cc:260] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.609062 32711 ts_tablet_manager.cc:1398] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:50.609968 32709 tablet_bootstrap.cc:654] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:50.612043 32709 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:50.612233 32709 ts_tablet_manager.cc:1367] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:50.612977 32709 raft_consensus.cc:348] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.613267 32709 raft_consensus.cc:374] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:50.613396 32709 raft_consensus.cc:729] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:50.613692 32709 consensus_queue.cc:260] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.614394 32709 ts_tablet_manager.cc:1398] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:50.629693 32481 raft_consensus.cc:482] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:50.629849 32481 raft_consensus.cc:504] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.630575 32481 leader_election.cc:288] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:50.630872 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:50.631129 28173 raft_consensus.cc:2454] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:50.631086 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:50.631281 28302 raft_consensus.cc:2454] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:50.632097 27977 leader_election.cc:302] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:50.635576 32481 raft_consensus.cc:2781] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:50.635665 32481 raft_consensus.cc:482] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:50.635753 32481 raft_consensus.cc:3037] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.639441 32481 raft_consensus.cc:504] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.640266 32481 leader_election.cc:288] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:50.640425 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:50.640501 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:50.640609 28173 raft_consensus.cc:3037] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.640692 28302 raft_consensus.cc:3037] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.642683 28173 raft_consensus.cc:2454] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:50.642760 28302 raft_consensus.cc:2454] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:50.643429 27977 leader_election.cc:302] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:50.644249 32481 raft_consensus.cc:2781] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:50.644507 32481 raft_consensus.cc:686] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:50.644798 32481 consensus_queue.cc:237] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:50.646593 32481 raft_consensus.cc:482] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:50.646728 32481 raft_consensus.cc:504] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.648203 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:50.648418 31070 raft_consensus.cc:2454] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:50.648857 27901 catalog_manager.cc:5484] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:50.649248 27978 leader_election.cc:302] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:50.649549 32689 raft_consensus.cc:2781] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:50.649641 32689 raft_consensus.cc:482] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:50.649724 32689 raft_consensus.cc:3037] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.652560 32689 raft_consensus.cc:504] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.653592 32689 leader_election.cc:288] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:50.655141 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:50.655364 28173 raft_consensus.cc:3037] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.662024 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:50.662222 31070 raft_consensus.cc:3037] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.665163 31070 raft_consensus.cc:2454] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:50.665555 27978 leader_election.cc:302] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:50.667685 28173 raft_consensus.cc:2454] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:50.667891 32723 raft_consensus.cc:2781] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:50.668042 32723 raft_consensus.cc:686] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:50.668381 32723 consensus_queue.cc:237] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:50.669787 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:50.670078 28173 raft_consensus.cc:2362] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1. Re-sending same reply.
I20240503 14:13:50.670814 32481 leader_election.cc:288] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:50.677210 27901 catalog_manager.cc:5484] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:50.704890 32723 raft_consensus.cc:482] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:50.705065 32723 raft_consensus.cc:504] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.705746 32723 leader_election.cc:288] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:50.706113 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd90464f4b84baabd90a44a122e28b1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:50.706391 31070 raft_consensus.cc:2454] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:50.706343 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd90464f4b84baabd90a44a122e28b1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:50.706540 30808 raft_consensus.cc:2454] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:50.706746 27978 leader_election.cc:302] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:50.707121 32723 raft_consensus.cc:2781] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:50.707211 32723 raft_consensus.cc:482] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:50.707326 32723 raft_consensus.cc:3037] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.709601 32723 raft_consensus.cc:504] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.710134 32723 leader_election.cc:288] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:50.710397 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd90464f4b84baabd90a44a122e28b1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:50.710587 31070 raft_consensus.cc:3037] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.713099 31070 raft_consensus.cc:2454] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:50.713550 27978 leader_election.cc:302] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:50.713797 32723 raft_consensus.cc:2781] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:50.713749 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd90464f4b84baabd90a44a122e28b1" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:50.713914 30808 raft_consensus.cc:3037] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.713960 32723 raft_consensus.cc:686] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:50.714198 32723 consensus_queue.cc:237] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:50.716215 30808 raft_consensus.cc:2454] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:50.717731 27901 catalog_manager.cc:5484] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:50.775269 32230 raft_consensus.cc:482] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:50.775467 32230 raft_consensus.cc:504] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.776471 32230 leader_election.cc:288] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:50.776865 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02180bd9a3704174a48ab282eecb619b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:50.777065 31203 raft_consensus.cc:2454] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:50.777065 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02180bd9a3704174a48ab282eecb619b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:50.777307 30808 raft_consensus.cc:2454] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:50.777632 28368 leader_election.cc:302] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:13:50.777922 32230 raft_consensus.cc:2781] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:50.778035 32230 raft_consensus.cc:482] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:50.778144 32230 raft_consensus.cc:3037] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.780169 32230 raft_consensus.cc:504] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.781257 32230 leader_election.cc:288] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:50.781324 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02180bd9a3704174a48ab282eecb619b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:50.781539 31203 raft_consensus.cc:3037] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.781164 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02180bd9a3704174a48ab282eecb619b" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:50.781702 30808 raft_consensus.cc:3037] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:50.783619 31203 raft_consensus.cc:2454] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:50.783730 30808 raft_consensus.cc:2454] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:50.784049 28368 leader_election.cc:302] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:13:50.784358 32230 raft_consensus.cc:2781] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:50.785208 32230 raft_consensus.cc:686] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:50.785646 32230 consensus_queue.cc:237] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:50.789469 27901 catalog_manager.cc:5484] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:50.933692 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:50.939265 31203 raft_consensus.cc:1261] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:50.939389 30808 raft_consensus.cc:1261] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:50.939762 32595 consensus_queue.cc:1035] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.940207 32230 consensus_queue.cc:1035] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:50.946183 32733 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:50.945062 27901 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 02180bd9a3704174a48ab282eecb619b (table rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005]). Expected version 1 got 0
I20240503 14:13:50.947417 32733 tablet.cc:1620] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:50.949826 32732 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:50.951022 32732 tablet.cc:1620] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:50.953974 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005] alter complete (version 1)
I20240503 14:13:50.963495 27898 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 02180bd9a3704174a48ab282eecb619b (table rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005]). Expected version 1 got 0
I20240503 14:13:50.965260 32735 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:50.965888 32735 tablet.cc:1620] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:50.969920 27901 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet 02180bd9a3704174a48ab282eecb619b (table rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005]). Expected version 1 got 0
I20240503 14:13:51.026502 32723 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.031466 32481 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.063508 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:51.073149 32732 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:51.073705 32732 tablet.cc:1620] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:51.074411 32735 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:51.074965 32735 tablet.cc:1620] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:51.084404 32733 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:51.085074 32733 tablet.cc:1620] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:51.086496 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=85b03bd803b641818407199aa51f8005] alter complete (version 2)
I20240503 14:13:51.109267 32723 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.113193 32723 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.180670 32481 consensus_queue.cc:1035] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.186568 32481 consensus_queue.cc:1035] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.191550 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:51.200973 32732 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:51.201768 32733 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:51.209862 32735 tablet.cc:1722] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:51.211102 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=85b03bd803b641818407199aa51f8005] alter complete (version 3)
I20240503 14:13:51.233196 27901 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:51.242627 31183 tablet_service.cc:1467] Processing CreateTablet for tablet b96b54a566db4bd1a31e8fb30096c942 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.243062 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96b54a566db4bd1a31e8fb30096c942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.243145 28412 tablet_service.cc:1467] Processing CreateTablet for tablet b96b54a566db4bd1a31e8fb30096c942 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.243556 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96b54a566db4bd1a31e8fb30096c942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.246644 30921 tablet_service.cc:1467] Processing CreateTablet for tablet b96b54a566db4bd1a31e8fb30096c942 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.247129 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96b54a566db4bd1a31e8fb30096c942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.253072 32749 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:51.253710 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 85a981e13c59422c87012c274884f9cf (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.254154 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a981e13c59422c87012c274884f9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.255383 30919 tablet_service.cc:1467] Processing CreateTablet for tablet a6b4ef3ee1fd440384272cee496fc102 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.255775 30919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b4ef3ee1fd440384272cee496fc102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.256700 32749 tablet_bootstrap.cc:654] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.258745 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 85a981e13c59422c87012c274884f9cf (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.259105 32748 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:51.259186 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a981e13c59422c87012c274884f9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.263582 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 85a981e13c59422c87012c274884f9cf (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.263993 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85a981e13c59422c87012c274884f9cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.263996 28283 tablet_service.cc:1467] Processing CreateTablet for tablet a6b4ef3ee1fd440384272cee496fc102 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.264415 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b4ef3ee1fd440384272cee496fc102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.267168 32748 tablet_bootstrap.cc:654] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.274187 32753 tablet_bootstrap.cc:492] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:51.274932 32752 tablet_bootstrap.cc:492] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:51.275661 31183 tablet_service.cc:1467] Processing CreateTablet for tablet a6b4ef3ee1fd440384272cee496fc102 (DEFAULT_TABLE table=rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:51.276166 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b4ef3ee1fd440384272cee496fc102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.276214 32753 tablet_bootstrap.cc:654] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.284183 32752 tablet_bootstrap.cc:654] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.286912 32755 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:51.288957 32747 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:51.289249 32755 tablet_bootstrap.cc:654] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.292043 32747 tablet_bootstrap.cc:654] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.293382 32748 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:51.293608 32748 ts_tablet_manager.cc:1367] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.009s
I20240503 14:13:51.295019 32748 raft_consensus.cc:348] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.295236 32752 tablet_bootstrap.cc:492] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:51.295377 32748 raft_consensus.cc:374] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.295409 32752 ts_tablet_manager.cc:1367] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240503 14:13:51.295557 32748 raft_consensus.cc:729] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.295879 32748 consensus_queue.cc:260] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.296180 32752 raft_consensus.cc:348] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.296391 32752 raft_consensus.cc:374] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.296514 32752 raft_consensus.cc:729] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.296972 32748 ts_tablet_manager.cc:1398] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:51.297976 32753 tablet_bootstrap.cc:492] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:51.298175 32753 ts_tablet_manager.cc:1367] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.004s
I20240503 14:13:51.298168 32752 consensus_queue.cc:260] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.298818 32752 ts_tablet_manager.cc:1398] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:51.299007 32753 raft_consensus.cc:348] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.299229 32753 raft_consensus.cc:374] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.299328 32753 raft_consensus.cc:729] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.299605 32753 consensus_queue.cc:260] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.300200 32753 ts_tablet_manager.cc:1398] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:51.300521 32748 tablet_bootstrap.cc:492] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:51.301537 32747 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:51.301748 32747 ts_tablet_manager.cc:1367] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240503 14:13:51.302501 32748 tablet_bootstrap.cc:654] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.302647 32747 raft_consensus.cc:348] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.302852 32747 raft_consensus.cc:374] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.302943 32747 raft_consensus.cc:729] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.303190 32747 consensus_queue.cc:260] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.303820 32747 ts_tablet_manager.cc:1398] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:51.299381 32755 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:51.311518 32755 ts_tablet_manager.cc:1367] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20240503 14:13:51.312310 32755 raft_consensus.cc:348] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.312525 32755 raft_consensus.cc:374] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.312618 32755 raft_consensus.cc:729] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.312860 32755 consensus_queue.cc:260] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.313987 32755 ts_tablet_manager.cc:1398] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:51.314196 32749 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:51.314421 32749 ts_tablet_manager.cc:1367] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.014s
I20240503 14:13:51.315801 32749 raft_consensus.cc:348] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.316069 32749 raft_consensus.cc:374] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.316216 32749 raft_consensus.cc:729] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.316448 32749 consensus_queue.cc:260] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.317101 32749 ts_tablet_manager.cc:1398] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:51.317426 32749 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:51.318125 27901 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:51.318346 27901 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:51.319677 32749 tablet_bootstrap.cc:654] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.324409 32748 tablet_bootstrap.cc:492] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:51.324591 32748 ts_tablet_manager.cc:1367] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.008s
I20240503 14:13:51.325249 32749 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:51.325407 32749 ts_tablet_manager.cc:1367] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 14:13:51.325327 32748 raft_consensus.cc:348] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.325558 32748 raft_consensus.cc:374] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.325646 32748 raft_consensus.cc:729] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.325883 32748 consensus_queue.cc:260] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.327101 32748 ts_tablet_manager.cc:1398] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:51.327433 32748 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:51.328115 32749 raft_consensus.cc:348] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.328343 32749 raft_consensus.cc:374] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.328416 32749 raft_consensus.cc:729] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.328636 32749 consensus_queue.cc:260] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.328927 32748 tablet_bootstrap.cc:654] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.330025 27901 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 02180bd9a3704174a48ab282eecb619b
I20240503 14:13:51.330483 32749 ts_tablet_manager.cc:1398] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:51.331110 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet 02180bd9a3704174a48ab282eecb619b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:51 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:51.331176 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 02180bd9a3704174a48ab282eecb619b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:51 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:51.331893 32762 tablet_replica.cc:330] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:51.332149 32762 raft_consensus.cc:2229] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.332559 32762 raft_consensus.cc:2258] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.333103 32763 tablet_replica.cc:330] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:51.333345 32763 raft_consensus.cc:2229] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:51.333760 32763 raft_consensus.cc:2258] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.334733 32763 ts_tablet_manager.cc:1875] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:51.337051 32762 ts_tablet_manager.cc:1875] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:51.339464 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 02180bd9a3704174a48ab282eecb619b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:51 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:51.343248 32748 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:51.343389 32748 ts_tablet_manager.cc:1367] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240503 14:13:51.344116 32748 raft_consensus.cc:348] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.344316 32748 raft_consensus.cc:374] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.344417 32748 raft_consensus.cc:729] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.344777 32762 ts_tablet_manager.cc:1888] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:51.344902 32762 log.cc:1196] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/02180bd9a3704174a48ab282eecb619b
I20240503 14:13:51.345307 32762 ts_tablet_manager.cc:1909] T 02180bd9a3704174a48ab282eecb619b P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:51.345857 32748 consensus_queue.cc:260] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.346326 32763 ts_tablet_manager.cc:1888] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:51.346544 32763 log.cc:1196] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/02180bd9a3704174a48ab282eecb619b
I20240503 14:13:51.347124 32763 ts_tablet_manager.cc:1909] T 02180bd9a3704174a48ab282eecb619b P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:51.347764 32748 ts_tablet_manager.cc:1398] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:51.349000 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 02180bd9a3704174a48ab282eecb619b (table rebalancer_dynamic_table_new_name [id=85b03bd803b641818407199aa51f8005]) successfully deleted
I20240503 14:13:51.353013 32641 raft_consensus.cc:482] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:51.353163 32641 raft_consensus.cc:504] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.354069 32641 leader_election.cc:288] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:51.355248 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a981e13c59422c87012c274884f9cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:51.355464 31070 raft_consensus.cc:2454] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:51.356098 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a981e13c59422c87012c274884f9cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:51.356298 31203 raft_consensus.cc:2454] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:51.356808 30745 leader_election.cc:302] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:51.357717 32641 raft_consensus.cc:2781] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:51.357831 32641 raft_consensus.cc:482] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:51.357908 32641 raft_consensus.cc:3037] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.358883 32764 tablet_replica.cc:330] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:51.359041 32764 raft_consensus.cc:2229] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.359217 32764 raft_consensus.cc:2258] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.361284 32641 raft_consensus.cc:504] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.362779 32764 ts_tablet_manager.cc:1875] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:51.362968 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a981e13c59422c87012c274884f9cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:51.363147 31203 raft_consensus.cc:3037] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.365957 31203 raft_consensus.cc:2454] T 85a981e13c59422c87012c274884f9cf P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:51.368371 30744 leader_election.cc:302] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:51.371099 32766 raft_consensus.cc:2781] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:51.371263 32766 raft_consensus.cc:686] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:51.372606 32766 consensus_queue.cc:237] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:51.375470 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85a981e13c59422c87012c274884f9cf" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:51.375676 31070 raft_consensus.cc:3037] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.377647 32764 ts_tablet_manager.cc:1888] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:51.377774 32764 log.cc:1196] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/02180bd9a3704174a48ab282eecb619b
I20240503 14:13:51.378225 32764 ts_tablet_manager.cc:1909] T 02180bd9a3704174a48ab282eecb619b P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:51.378445 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 02180bd9a3704174a48ab282eecb619b (table rebalancer_dynamic_table_new_name [id=85b03bd803b641818407199aa51f8005]) successfully deleted
I20240503 14:13:51.379330 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 02180bd9a3704174a48ab282eecb619b (table rebalancer_dynamic_table_new_name [id=85b03bd803b641818407199aa51f8005]) successfully deleted
I20240503 14:13:51.380882 31070 raft_consensus.cc:2454] T 85a981e13c59422c87012c274884f9cf P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:51.381445 32641 leader_election.cc:288] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:51.395817 27899 catalog_manager.cc:5484] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:51.440570 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:51.449033 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 159e332692084e9ca11cb1e47948507c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f]), partition=
I20240503 14:13:51.449201 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 159e332692084e9ca11cb1e47948507c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f]), partition=
I20240503 14:13:51.449391 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 159e332692084e9ca11cb1e47948507c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f]), partition=
I20240503 14:13:51.449478 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159e332692084e9ca11cb1e47948507c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.449502 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159e332692084e9ca11cb1e47948507c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.449707 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 159e332692084e9ca11cb1e47948507c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:51.453572 32747 tablet_bootstrap.cc:492] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:51.455001 32747 tablet_bootstrap.cc:654] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.457085 32747 tablet_bootstrap.cc:492] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:51.457254 32747 ts_tablet_manager.cc:1367] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:51.458453 32752 tablet_bootstrap.cc:492] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:51.459182 32755 tablet_bootstrap.cc:492] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:51.459290 32747 raft_consensus.cc:348] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.459503 32747 raft_consensus.cc:374] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.459574 32747 raft_consensus.cc:729] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.459735 32752 tablet_bootstrap.cc:654] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.460553 32747 consensus_queue.cc:260] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.460667 32755 tablet_bootstrap.cc:654] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:51.461011 32747 ts_tablet_manager.cc:1398] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:51.462270 32752 tablet_bootstrap.cc:492] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:51.462424 32752 ts_tablet_manager.cc:1367] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:51.462553 32755 tablet_bootstrap.cc:492] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:51.462700 32755 ts_tablet_manager.cc:1367] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:51.463109 32752 raft_consensus.cc:348] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.463341 32752 raft_consensus.cc:374] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.463423 32752 raft_consensus.cc:729] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.463415 32755 raft_consensus.cc:348] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.463690 32752 consensus_queue.cc:260] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.463925 32755 raft_consensus.cc:374] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:51.463999 32755 raft_consensus.cc:729] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:51.464231 32752 ts_tablet_manager.cc:1398] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:51.464181 32755 consensus_queue.cc:260] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.471778 32755 ts_tablet_manager.cc:1398] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:51.476538 32636 raft_consensus.cc:482] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:51.476743 32636 raft_consensus.cc:504] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.477551 32636 leader_election.cc:288] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:51.477874 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ef3ee1fd440384272cee496fc102" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:51.477989 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ef3ee1fd440384272cee496fc102" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:51.478071 28302 raft_consensus.cc:2454] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:51.478227 31203 raft_consensus.cc:2454] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:51.478451 30877 leader_election.cc:302] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:51.478822 32636 raft_consensus.cc:2781] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:51.478910 32636 raft_consensus.cc:482] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:51.479014 32636 raft_consensus.cc:3037] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.481338 32636 raft_consensus.cc:504] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.481940 32636 leader_election.cc:288] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:51.482322 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ef3ee1fd440384272cee496fc102" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:51.482347 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6b4ef3ee1fd440384272cee496fc102" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:51.482525 28302 raft_consensus.cc:3037] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.482561 31203 raft_consensus.cc:3037] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.484527 28302 raft_consensus.cc:2454] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:51.484894 30877 leader_election.cc:302] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:51.485168 32636 raft_consensus.cc:2781] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:51.485383 32636 raft_consensus.cc:686] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:51.485821 31203 raft_consensus.cc:2454] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:51.486001 32636 consensus_queue.cc:237] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:51.491421 27901 catalog_manager.cc:5484] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:51.538345 32230 raft_consensus.cc:482] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:51.538553 32230 raft_consensus.cc:504] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.539149 32230 leader_election.cc:288] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:51.539515 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b54a566db4bd1a31e8fb30096c942" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:51.539719 31203 raft_consensus.cc:2454] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:51.539693 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b54a566db4bd1a31e8fb30096c942" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:51.539932 30940 raft_consensus.cc:2454] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:51.540097 28368 leader_election.cc:302] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:13:51.540426 32230 raft_consensus.cc:2781] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:51.540562 32230 raft_consensus.cc:482] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:51.540664 32230 raft_consensus.cc:3037] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.543514 32230 raft_consensus.cc:504] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.544152 32230 leader_election.cc:288] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:51.544636 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b54a566db4bd1a31e8fb30096c942" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:51.544762 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b96b54a566db4bd1a31e8fb30096c942" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:51.544907 30940 raft_consensus.cc:3037] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.544973 31203 raft_consensus.cc:3037] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.546061 32661 raft_consensus.cc:482] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:51.546339 32661 raft_consensus.cc:504] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.547596 32661 leader_election.cc:288] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:51.547648 31203 raft_consensus.cc:2454] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:51.547794 30940 raft_consensus.cc:2454] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:51.548010 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159e332692084e9ca11cb1e47948507c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:51.548209 28369 leader_election.cc:302] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:51.548290 31070 raft_consensus.cc:2454] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:51.548717 28239 leader_election.cc:302] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:51.548995 32638 raft_consensus.cc:2781] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:51.549093 32638 raft_consensus.cc:482] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:51.549091 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159e332692084e9ca11cb1e47948507c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:51.549198 32638 raft_consensus.cc:3037] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.549043 32230 raft_consensus.cc:2781] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:51.549317 28432 raft_consensus.cc:2454] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:51.549351 32230 raft_consensus.cc:686] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:51.549849 32230 consensus_queue.cc:237] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:51.552999 32638 raft_consensus.cc:504] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.554397 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159e332692084e9ca11cb1e47948507c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:51.554659 31070 raft_consensus.cc:3037] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.554626 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "159e332692084e9ca11cb1e47948507c" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:51.554805 28432 raft_consensus.cc:3037] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:51.557180 31070 raft_consensus.cc:2454] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:51.557549 28239 leader_election.cc:302] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:51.557400 28432 raft_consensus.cc:2454] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:51.557777 32661 raft_consensus.cc:2781] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:51.558352 32638 leader_election.cc:288] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:51.559293 27901 catalog_manager.cc:5484] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:51.560452 32661 raft_consensus.cc:686] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:51.560757 32661 consensus_queue.cc:237] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:51.565663 27901 catalog_manager.cc:5484] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:51.675742 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:51.682001 28432 raft_consensus.cc:1261] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:51.682183 31070 raft_consensus.cc:1261] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:51.682791 308 consensus_queue.cc:1035] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.683851 308 consensus_queue.cc:1035] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.689939 27899 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 159e332692084e9ca11cb1e47948507c (table rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f]). Expected version 1 got 0
I20240503 14:13:51.691390 32732 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:51.691951 32732 tablet.cc:1620] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:51.695047 27899 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 159e332692084e9ca11cb1e47948507c (table rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f]). Expected version 1 got 0
I20240503 14:13:51.698626 27901 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 159e332692084e9ca11cb1e47948507c (table rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f]). Expected version 1 got 0
I20240503 14:13:51.698627 314 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:51.700769 314 tablet.cc:1620] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:51.704484 315 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:51.705773 315 tablet.cc:1620] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:51.708168 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f] alter complete (version 1)
I20240503 14:13:51.709370 27899 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet 159e332692084e9ca11cb1e47948507c (table rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f]). Expected version 1 got 0
I20240503 14:13:51.816212 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:51.826435 314 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:51.826910 314 tablet.cc:1620] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:51.831400 315 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:51.831888 315 tablet.cc:1620] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:51.839027 32732 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:51.843653 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=75312c494c6848aaa535e32ba6b15a5f] alter complete (version 2)
I20240503 14:13:51.844218 32732 tablet.cc:1620] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:51.877537 32766 consensus_queue.cc:1035] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.890525 32767 consensus_queue.cc:1035] T 85a981e13c59422c87012c274884f9cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.912536 32635 consensus_queue.cc:1035] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.947230 32636 consensus_queue.cc:1035] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.948875 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:51.968072 314 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:51.969457 32732 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:51.986426 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=75312c494c6848aaa535e32ba6b15a5f] alter complete (version 3)
I20240503 14:13:51.987540 32230 consensus_queue.cc:1035] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:51.995496 315 tablet.cc:1722] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:51.996699 32595 consensus_queue.cc:1035] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.091611 27898 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:52.091889 27898 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:52.094952 27898 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 159e332692084e9ca11cb1e47948507c
I20240503 14:13:52.095772 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet 159e332692084e9ca11cb1e47948507c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:52 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:52.095784 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 159e332692084e9ca11cb1e47948507c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:52 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:52.096159 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 159e332692084e9ca11cb1e47948507c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:52 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:52.096448 327 tablet_replica.cc:330] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:52.096732 327 raft_consensus.cc:2229] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:52.096853 326 tablet_replica.cc:330] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:52.097158 326 raft_consensus.cc:2229] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:52.097179 327 raft_consensus.cc:2258] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.097393 328 tablet_replica.cc:330] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:52.097385 326 raft_consensus.cc:2258] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.097620 328 raft_consensus.cc:2229] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:52.097862 328 raft_consensus.cc:2258] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.097954 327 ts_tablet_manager.cc:1875] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:52.098526 326 ts_tablet_manager.cc:1875] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:52.098868 328 ts_tablet_manager.cc:1875] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:52.103535 327 ts_tablet_manager.cc:1888] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:52.103654 327 log.cc:1196] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/159e332692084e9ca11cb1e47948507c
I20240503 14:13:52.104038 327 ts_tablet_manager.cc:1909] T 159e332692084e9ca11cb1e47948507c P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:52.104876 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 159e332692084e9ca11cb1e47948507c (table rebalancer_dynamic_table_new_name [id=75312c494c6848aaa535e32ba6b15a5f]) successfully deleted
I20240503 14:13:52.106071 328 ts_tablet_manager.cc:1888] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:52.106251 326 ts_tablet_manager.cc:1888] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:52.106277 328 log.cc:1196] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/159e332692084e9ca11cb1e47948507c
I20240503 14:13:52.106392 326 log.cc:1196] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/159e332692084e9ca11cb1e47948507c
I20240503 14:13:52.106750 328 ts_tablet_manager.cc:1909] T 159e332692084e9ca11cb1e47948507c P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:52.106876 326 ts_tablet_manager.cc:1909] T 159e332692084e9ca11cb1e47948507c P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:52.107790 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 159e332692084e9ca11cb1e47948507c (table rebalancer_dynamic_table_new_name [id=75312c494c6848aaa535e32ba6b15a5f]) successfully deleted
I20240503 14:13:52.108024 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 159e332692084e9ca11cb1e47948507c (table rebalancer_dynamic_table_new_name [id=75312c494c6848aaa535e32ba6b15a5f]) successfully deleted
I20240503 14:13:52.127097 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:52.136824 28283 tablet_service.cc:1467] Processing CreateTablet for tablet e38e89746d9944e5a12f6b68a0ee13db (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.137259 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38e89746d9944e5a12f6b68a0ee13db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.137174 30790 tablet_service.cc:1467] Processing CreateTablet for tablet e38e89746d9944e5a12f6b68a0ee13db (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.137226 28021 tablet_service.cc:1467] Processing CreateTablet for tablet e38e89746d9944e5a12f6b68a0ee13db (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.137586 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38e89746d9944e5a12f6b68a0ee13db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.137595 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e38e89746d9944e5a12f6b68a0ee13db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.142266 329 tablet_bootstrap.cc:492] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:52.144096 30789 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4fbfaadc474d59bfb23107f0342f7c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.144309 329 tablet_bootstrap.cc:654] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.144282 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4fbfaadc474d59bfb23107f0342f7c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.144595 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4fbfaadc474d59bfb23107f0342f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.144737 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4fbfaadc474d59bfb23107f0342f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.151819 28023 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4fbfaadc474d59bfb23107f0342f7c (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.152740 30788 tablet_service.cc:1467] Processing CreateTablet for tablet c761d663c0bd48529edc9e454d5292ac (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.153669 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4fbfaadc474d59bfb23107f0342f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.153285 30788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c761d663c0bd48529edc9e454d5292ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.162650 31051 tablet_service.cc:1467] Processing CreateTablet for tablet c761d663c0bd48529edc9e454d5292ac (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.163255 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c761d663c0bd48529edc9e454d5292ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.163909 28154 tablet_service.cc:1467] Processing CreateTablet for tablet c761d663c0bd48529edc9e454d5292ac (DEFAULT_TABLE table=rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:52.164338 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c761d663c0bd48529edc9e454d5292ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.183923 335 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:52.186049 332 tablet_bootstrap.cc:492] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:52.188388 333 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:52.188839 334 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:52.190820 335 tablet_bootstrap.cc:654] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.193090 332 tablet_bootstrap.cc:654] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.193388 333 tablet_bootstrap.cc:654] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.194804 334 tablet_bootstrap.cc:654] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.195843 329 tablet_bootstrap.cc:492] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:52.196089 329 ts_tablet_manager.cc:1367] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.017s
I20240503 14:13:52.197705 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:52.197901 329 raft_consensus.cc:348] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.198262 329 raft_consensus.cc:374] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.198455 329 raft_consensus.cc:729] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.198860 329 consensus_queue.cc:260] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.202220 329 ts_tablet_manager.cc:1398] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:52.210285 332 tablet_bootstrap.cc:492] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:52.210553 332 ts_tablet_manager.cc:1367] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.008s
I20240503 14:13:52.211431 28154 tablet_service.cc:1467] Processing CreateTablet for tablet b1c583dfe73345a380ab05e1e0c7e3a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651]), partition=
I20240503 14:13:52.211777 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c583dfe73345a380ab05e1e0c7e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.211982 335 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:52.212114 335 ts_tablet_manager.cc:1367] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240503 14:13:52.212966 335 raft_consensus.cc:348] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.213218 335 raft_consensus.cc:374] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.213253 31183 tablet_service.cc:1467] Processing CreateTablet for tablet b1c583dfe73345a380ab05e1e0c7e3a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651]), partition=
I20240503 14:13:52.213328 335 raft_consensus.cc:729] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.213588 335 consensus_queue.cc:260] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.213706 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c583dfe73345a380ab05e1e0c7e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.217876 335 ts_tablet_manager.cc:1398] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:52.230711 30789 tablet_service.cc:1467] Processing CreateTablet for tablet b1c583dfe73345a380ab05e1e0c7e3a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651]), partition=
I20240503 14:13:52.230710 332 raft_consensus.cc:348] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.230979 332 raft_consensus.cc:374] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.231122 332 raft_consensus.cc:729] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.231397 332 consensus_queue.cc:260] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.232528 332 ts_tablet_manager.cc:1398] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20240503 14:13:52.232865 332 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:52.234566 332 tablet_bootstrap.cc:654] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.252759 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c583dfe73345a380ab05e1e0c7e3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:52.263250 339 tablet_bootstrap.cc:492] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:52.265592 339 tablet_bootstrap.cc:654] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.270603 333 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:52.270654 330 tablet_bootstrap.cc:492] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:52.270737 334 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:52.270906 333 ts_tablet_manager.cc:1367] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.000s
I20240503 14:13:52.271862 334 ts_tablet_manager.cc:1367] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20240503 14:13:52.271749 333 raft_consensus.cc:348] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.271984 333 raft_consensus.cc:374] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.272086 333 raft_consensus.cc:729] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.272310 333 consensus_queue.cc:260] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.272733 333 ts_tablet_manager.cc:1398] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:52.273340 330 tablet_bootstrap.cc:654] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.274293 334 raft_consensus.cc:348] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.273018 333 tablet_bootstrap.cc:492] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:52.274637 334 raft_consensus.cc:374] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.274792 334 raft_consensus.cc:729] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.275158 334 consensus_queue.cc:260] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.275527 332 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:52.276216 332 ts_tablet_manager.cc:1367] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.010s
I20240503 14:13:52.277269 332 raft_consensus.cc:348] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.279707 334 ts_tablet_manager.cc:1398] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 14:13:52.284569 330 tablet_bootstrap.cc:492] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:52.284744 330 ts_tablet_manager.cc:1367] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
I20240503 14:13:52.285725 330 raft_consensus.cc:348] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.285926 330 raft_consensus.cc:374] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.286027 330 raft_consensus.cc:729] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.286299 332 raft_consensus.cc:374] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.286316 330 consensus_queue.cc:260] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.286442 332 raft_consensus.cc:729] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.286871 330 ts_tablet_manager.cc:1398] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:52.287194 330 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:52.287431 332 consensus_queue.cc:260] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.289165 332 ts_tablet_manager.cc:1398] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 14:13:52.290437 330 tablet_bootstrap.cc:654] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.291786 333 tablet_bootstrap.cc:654] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.294142 339 tablet_bootstrap.cc:492] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:52.294381 339 ts_tablet_manager.cc:1367] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.000s
I20240503 14:13:52.295552 339 raft_consensus.cc:348] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.295720 339 raft_consensus.cc:374] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.295851 339 raft_consensus.cc:729] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.296105 32766 raft_consensus.cc:482] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:52.296326 339 consensus_queue.cc:260] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.296308 32766 raft_consensus.cc:504] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.297266 32766 leader_election.cc:288] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:52.297756 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e89746d9944e5a12f6b68a0ee13db" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:52.297966 28302 raft_consensus.cc:2454] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:52.297919 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e89746d9944e5a12f6b68a0ee13db" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:52.298116 28042 raft_consensus.cc:2454] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:52.298614 30746 leader_election.cc:302] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:52.298794 339 ts_tablet_manager.cc:1398] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:13:52.299621 333 tablet_bootstrap.cc:492] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:52.299782 333 ts_tablet_manager.cc:1367] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20240503 14:13:52.300354 333 raft_consensus.cc:348] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.300523 333 raft_consensus.cc:374] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.300607 333 raft_consensus.cc:729] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.300841 333 consensus_queue.cc:260] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.301368 333 ts_tablet_manager.cc:1398] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:13:52.302371 32767 raft_consensus.cc:2781] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:52.302506 32767 raft_consensus.cc:482] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:52.302613 32767 raft_consensus.cc:3037] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.307343 32767 raft_consensus.cc:504] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.308476 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e89746d9944e5a12f6b68a0ee13db" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:52.308656 28302 raft_consensus.cc:3037] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.310483 32767 leader_election.cc:288] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:52.310638 28302 raft_consensus.cc:2454] T e38e89746d9944e5a12f6b68a0ee13db P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:52.311107 30746 leader_election.cc:302] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:52.311509 32766 raft_consensus.cc:2781] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:52.311471 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38e89746d9944e5a12f6b68a0ee13db" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:52.311643 32766 raft_consensus.cc:686] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:52.311661 28042 raft_consensus.cc:3037] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.311873 32766 consensus_queue.cc:237] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.316596 28042 raft_consensus.cc:2454] T e38e89746d9944e5a12f6b68a0ee13db P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:52.318675 330 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:52.318902 330 ts_tablet_manager.cc:1367] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.019s
I20240503 14:13:52.320073 330 raft_consensus.cc:348] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.320297 330 raft_consensus.cc:374] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.320380 330 raft_consensus.cc:729] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.320611 330 consensus_queue.cc:260] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.321022 330 ts_tablet_manager.cc:1398] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:52.321254 330 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:52.322643 330 tablet_bootstrap.cc:654] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.324553 330 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:52.324710 330 ts_tablet_manager.cc:1367] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:52.325752 330 raft_consensus.cc:348] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.326002 330 raft_consensus.cc:374] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.326099 330 raft_consensus.cc:729] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.326392 330 consensus_queue.cc:260] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.326969 330 ts_tablet_manager.cc:1398] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:52.327267 330 tablet_bootstrap.cc:492] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:52.328984 330 tablet_bootstrap.cc:654] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:52.331022 330 tablet_bootstrap.cc:492] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:52.331209 330 ts_tablet_manager.cc:1367] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:52.331861 330 raft_consensus.cc:348] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.332089 330 raft_consensus.cc:374] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:52.332183 330 raft_consensus.cc:729] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:52.332388 330 consensus_queue.cc:260] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.332955 330 ts_tablet_manager.cc:1398] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:52.333640 27901 catalog_manager.cc:5484] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:52.427824 301 raft_consensus.cc:482] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:52.427989 301 raft_consensus.cc:504] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.429100 301 leader_election.cc:288] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:52.429384 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c761d663c0bd48529edc9e454d5292ac" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:52.429579 30809 raft_consensus.cc:2454] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:52.429564 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c761d663c0bd48529edc9e454d5292ac" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:52.429773 28174 raft_consensus.cc:2454] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:52.430003 31005 leader_election.cc:302] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:52.430300 301 raft_consensus.cc:2781] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:52.430460 301 raft_consensus.cc:482] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:52.430562 301 raft_consensus.cc:3037] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.430944 316 raft_consensus.cc:482] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:52.431057 316 raft_consensus.cc:504] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.432197 316 leader_election.cc:288] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:52.432767 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c583dfe73345a380ab05e1e0c7e3a3" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:52.432785 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c583dfe73345a380ab05e1e0c7e3a3" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:52.432996 30809 raft_consensus.cc:2454] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:52.433028 31203 raft_consensus.cc:2454] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:52.433287 301 raft_consensus.cc:504] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.433405 28107 leader_election.cc:302] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:52.433655 316 raft_consensus.cc:2781] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:52.433766 316 raft_consensus.cc:482] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:52.433867 316 raft_consensus.cc:3037] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.434489 301 leader_election.cc:288] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:52.435653 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c761d663c0bd48529edc9e454d5292ac" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:52.435907 30809 raft_consensus.cc:3037] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.436883 316 raft_consensus.cc:504] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.437587 316 leader_election.cc:288] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:52.437927 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c583dfe73345a380ab05e1e0c7e3a3" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:52.438113 30808 raft_consensus.cc:3037] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.438072 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1c583dfe73345a380ab05e1e0c7e3a3" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:52.438105 28174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c761d663c0bd48529edc9e454d5292ac" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:52.438308 28174 raft_consensus.cc:3037] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.438297 31203 raft_consensus.cc:3037] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.440095 31203 raft_consensus.cc:2454] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:52.440513 28108 leader_election.cc:302] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:52.440863 28174 raft_consensus.cc:2454] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:52.440855 30808 raft_consensus.cc:2454] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:52.441120 316 raft_consensus.cc:2781] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:52.441591 316 raft_consensus.cc:686] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:52.441956 31006 leader_election.cc:302] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:52.441892 316 consensus_queue.cc:237] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:52.442864 301 raft_consensus.cc:2781] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:52.444569 30809 raft_consensus.cc:2454] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:52.446452 301 raft_consensus.cc:686] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:52.446367 27894 catalog_manager.cc:5484] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:52.446750 301 consensus_queue.cc:237] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:52.450173 27901 catalog_manager.cc:5484] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:52.473495 32635 raft_consensus.cc:482] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:52.473697 32635 raft_consensus.cc:504] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.474427 32635 leader_election.cc:288] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:52.474807 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4fbfaadc474d59bfb23107f0342f7c" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:52.475035 30809 raft_consensus.cc:2454] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:52.475127 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4fbfaadc474d59bfb23107f0342f7c" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:52.475374 28040 raft_consensus.cc:2454] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:52.475430 30874 leader_election.cc:302] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:52.475809 32635 raft_consensus.cc:2781] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:52.475914 32635 raft_consensus.cc:482] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:52.476024 32635 raft_consensus.cc:3037] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.477993 32635 raft_consensus.cc:504] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.478632 32635 leader_election.cc:288] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:52.479346 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4fbfaadc474d59bfb23107f0342f7c" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:52.479558 30809 raft_consensus.cc:3037] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.479552 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4fbfaadc474d59bfb23107f0342f7c" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:52.479786 28040 raft_consensus.cc:3037] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:52.482548 28040 raft_consensus.cc:2454] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:52.483014 30877 leader_election.cc:302] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:52.483299 32635 raft_consensus.cc:2781] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:52.483460 32635 raft_consensus.cc:686] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:52.483826 32635 consensus_queue.cc:237] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:52.484753 30809 raft_consensus.cc:2454] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:52.488837 27901 catalog_manager.cc:5484] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:52.563647 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:52.569507 31202 raft_consensus.cc:1261] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:52.569510 30809 raft_consensus.cc:1261] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:52.570071 316 consensus_queue.cc:1035] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.570343 32601 consensus_queue.cc:1035] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.576470 27901 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet b1c583dfe73345a380ab05e1e0c7e3a3 (table rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651]). Expected version 1 got 0
I20240503 14:13:52.577742 360 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:52.578536 360 tablet.cc:1620] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:52.579772 361 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:52.580503 361 tablet.cc:1620] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:52.582468 27901 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet b1c583dfe73345a380ab05e1e0c7e3a3 (table rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651]). Expected version 1 got 0
I20240503 14:13:52.589896 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651] alter complete (version 1)
I20240503 14:13:52.594969 362 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:52.595561 362 tablet.cc:1620] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:52.597206 27901 catalog_manager.cc:5576] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689) does not have the latest schema for tablet b1c583dfe73345a380ab05e1e0c7e3a3 (table rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651]). Expected version 1 got 0
I20240503 14:13:52.695312 27901 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:52.702689 360 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:52.703227 360 tablet.cc:1620] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:52.703538 361 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:52.704001 361 tablet.cc:1620] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:52.711000 362 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:52.711450 362 tablet.cc:1620] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: MemRowSet was empty: no flush needed.
I20240503 14:13:52.714483 27901 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=116f4956c5bd4695b049cbcfde802651] alter complete (version 2)
I20240503 14:13:52.719933 32766 consensus_queue.cc:1035] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.724530 32766 consensus_queue.cc:1035] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.816959 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:52.824834 360 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:52.825575 361 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:52.825724 362 tablet.cc:1722] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:52.836330 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=116f4956c5bd4695b049cbcfde802651] alter complete (version 3)
I20240503 14:13:52.938632 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:52.938843 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:52.942085 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet b1c583dfe73345a380ab05e1e0c7e3a3
I20240503 14:13:52.942108 364 consensus_queue.cc:1035] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.942819 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c583dfe73345a380ab05e1e0c7e3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:52 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:52.943219 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c583dfe73345a380ab05e1e0c7e3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:52 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:52.943411 30790 tablet_service.cc:1514] Processing DeleteTablet for tablet b1c583dfe73345a380ab05e1e0c7e3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:52 UTC) from {username='slave'} at 127.0.0.1:48954
I20240503 14:13:52.944561 371 tablet_replica.cc:330] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:52.944914 371 raft_consensus.cc:2229] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:52.945438 371 raft_consensus.cc:2258] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.946295 372 tablet_replica.cc:330] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:52.946666 372 raft_consensus.cc:2229] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:52.946763 371 ts_tablet_manager.cc:1875] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:52.947021 372 raft_consensus.cc:2258] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.948040 372 ts_tablet_manager.cc:1875] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:52.955235 371 ts_tablet_manager.cc:1888] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:52.955415 371 log.cc:1196] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/b1c583dfe73345a380ab05e1e0c7e3a3
I20240503 14:13:52.955940 371 ts_tablet_manager.cc:1909] T b1c583dfe73345a380ab05e1e0c7e3a3 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:52.955986 301 consensus_queue.cc:1035] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.957093 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet b1c583dfe73345a380ab05e1e0c7e3a3 (table rebalancer_dynamic_table_new_name [id=116f4956c5bd4695b049cbcfde802651]) successfully deleted
I20240503 14:13:52.958177 373 tablet_replica.cc:330] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:13:52.958709 372 ts_tablet_manager.cc:1888] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:52.958907 372 log.cc:1196] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/b1c583dfe73345a380ab05e1e0c7e3a3
I20240503 14:13:52.958987 373 raft_consensus.cc:2229] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:52.959448 372 ts_tablet_manager.cc:1909] T b1c583dfe73345a380ab05e1e0c7e3a3 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:52.959570 373 raft_consensus.cc:2258] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.960557 373 ts_tablet_manager.cc:1875] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:52.962222 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet b1c583dfe73345a380ab05e1e0c7e3a3 (table rebalancer_dynamic_table_new_name [id=116f4956c5bd4695b049cbcfde802651]) successfully deleted
I20240503 14:13:52.972851 32635 consensus_queue.cc:1035] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.974959 373 ts_tablet_manager.cc:1888] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:52.975235 373 log.cc:1196] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/b1c583dfe73345a380ab05e1e0c7e3a3
I20240503 14:13:52.976372 373 ts_tablet_manager.cc:1909] T b1c583dfe73345a380ab05e1e0c7e3a3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting consensus metadata
I20240503 14:13:52.978646 301 consensus_queue.cc:1035] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:52.982658 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet b1c583dfe73345a380ab05e1e0c7e3a3 (table rebalancer_dynamic_table_new_name [id=116f4956c5bd4695b049cbcfde802651]) successfully deleted
I20240503 14:13:53.029821 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:53.039319 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 16c2801d0bf442519ab3841a98d37696 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.039813 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c2801d0bf442519ab3841a98d37696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.044452 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:53.046648 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 16c2801d0bf442519ab3841a98d37696 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.047129 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c2801d0bf442519ab3841a98d37696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.047034 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 16c2801d0bf442519ab3841a98d37696 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.047472 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c2801d0bf442519ab3841a98d37696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.050618 31051 tablet_service.cc:1467] Processing CreateTablet for tablet f6cbcf7c01bb4265a7c8fd4c897c4928 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.051002 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cbcf7c01bb4265a7c8fd4c897c4928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.054613 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac53dcb19f941c48dd6c6cb623ccf58 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.054625 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac53dcb19f941c48dd6c6cb623ccf58 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.055013 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac53dcb19f941c48dd6c6cb623ccf58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.055029 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac53dcb19f941c48dd6c6cb623ccf58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.058903 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac53dcb19f941c48dd6c6cb623ccf58 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.059307 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac53dcb19f941c48dd6c6cb623ccf58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.059381 28414 tablet_service.cc:1467] Processing CreateTablet for tablet f6cbcf7c01bb4265a7c8fd4c897c4928 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.059860 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cbcf7c01bb4265a7c8fd4c897c4928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.063308 30790 tablet_service.cc:1467] Processing CreateTablet for tablet f6cbcf7c01bb4265a7c8fd4c897c4928 (DEFAULT_TABLE table=rebalancer_extra_table_60 [id=35a467cfeb2e4be191537ff95add7ace]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:53.063845 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6cbcf7c01bb4265a7c8fd4c897c4928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.073632 28413 tablet_service.cc:1467] Processing CreateTablet for tablet 7c487a3f48f849298b3aec3155a559e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1]), partition=
I20240503 14:13:53.073616 31182 tablet_service.cc:1467] Processing CreateTablet for tablet 7c487a3f48f849298b3aec3155a559e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1]), partition=
I20240503 14:13:53.073999 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c487a3f48f849298b3aec3155a559e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.074313 28282 tablet_service.cc:1467] Processing CreateTablet for tablet 7c487a3f48f849298b3aec3155a559e4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1]), partition=
I20240503 14:13:53.074632 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c487a3f48f849298b3aec3155a559e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.083774 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c487a3f48f849298b3aec3155a559e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.093129 384 tablet_bootstrap.cc:492] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:53.093813 386 tablet_bootstrap.cc:492] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:53.103053 383 tablet_bootstrap.cc:492] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:53.104154 384 tablet_bootstrap.cc:654] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.110837 384 tablet_bootstrap.cc:492] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:53.111905 384 ts_tablet_manager.cc:1367] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240503 14:13:53.112340 383 tablet_bootstrap.cc:654] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.113656 384 raft_consensus.cc:348] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.116956 386 tablet_bootstrap.cc:654] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.117651 387 tablet_bootstrap.cc:492] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:53.118146 381 tablet_bootstrap.cc:492] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:53.118721 384 raft_consensus.cc:374] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.119576 384 raft_consensus.cc:729] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.121034 385 tablet_bootstrap.cc:492] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:53.121409 384 consensus_queue.cc:260] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.121717 389 tablet_bootstrap.cc:492] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:53.122947 384 ts_tablet_manager.cc:1398] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:13:53.123750 389 tablet_bootstrap.cc:654] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.124574 381 tablet_bootstrap.cc:654] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.128844 387 tablet_bootstrap.cc:654] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.132155 382 tablet_bootstrap.cc:492] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:53.136225 382 tablet_bootstrap.cc:654] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.138013 385 tablet_bootstrap.cc:654] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.147984 385 tablet_bootstrap.cc:492] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:53.149386 386 tablet_bootstrap.cc:492] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:53.150077 387 tablet_bootstrap.cc:492] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:53.150631 383 tablet_bootstrap.cc:492] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:53.150840 382 tablet_bootstrap.cc:492] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:53.150835 383 ts_tablet_manager.cc:1367] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.003s
I20240503 14:13:53.151058 387 ts_tablet_manager.cc:1367] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240503 14:13:53.151685 383 raft_consensus.cc:348] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.151813 387 raft_consensus.cc:348] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.152161 385 ts_tablet_manager.cc:1367] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240503 14:13:53.152241 383 raft_consensus.cc:374] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.152367 386 ts_tablet_manager.cc:1367] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.004s
I20240503 14:13:53.152396 383 raft_consensus.cc:729] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.152752 383 consensus_queue.cc:260] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.153254 385 raft_consensus.cc:348] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.153566 382 ts_tablet_manager.cc:1367] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 14:13:53.154628 382 raft_consensus.cc:348] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.155440 386 raft_consensus.cc:348] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.155717 387 raft_consensus.cc:374] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.155761 383 ts_tablet_manager.cc:1398] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:53.155897 387 raft_consensus.cc:729] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.156140 387 consensus_queue.cc:260] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.156355 385 raft_consensus.cc:374] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.156455 385 raft_consensus.cc:729] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.156773 385 consensus_queue.cc:260] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.157047 386 raft_consensus.cc:374] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.157183 382 raft_consensus.cc:374] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.157333 386 raft_consensus.cc:729] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.159922 387 ts_tablet_manager.cc:1398] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 14:13:53.160092 382 raft_consensus.cc:729] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.160408 386 consensus_queue.cc:260] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.160892 382 consensus_queue.cc:260] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.161521 385 ts_tablet_manager.cc:1398] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:53.163540 382 ts_tablet_manager.cc:1398] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:13:53.163820 381 tablet_bootstrap.cc:492] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:53.163983 386 ts_tablet_manager.cc:1398] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:13:53.163980 381 ts_tablet_manager.cc:1367] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.022s
I20240503 14:13:53.164853 381 raft_consensus.cc:348] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.165052 381 raft_consensus.cc:374] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.165134 381 raft_consensus.cc:729] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.165369 381 consensus_queue.cc:260] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.165951 381 ts_tablet_manager.cc:1398] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:53.169705 386 tablet_bootstrap.cc:492] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:53.170135 353 raft_consensus.cc:482] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:53.170316 353 raft_consensus.cc:504] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.171078 389 tablet_bootstrap.cc:492] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:53.171142 353 leader_election.cc:288] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:53.171319 389 ts_tablet_manager.cc:1367] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.003s
I20240503 14:13:53.171420 381 tablet_bootstrap.cc:492] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:53.172260 389 raft_consensus.cc:348] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.172591 389 raft_consensus.cc:374] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.172740 389 raft_consensus.cc:729] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.173063 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c2801d0bf442519ab3841a98d37696" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:53.173081 389 consensus_queue.cc:260] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.173334 30939 raft_consensus.cc:2454] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:53.173736 381 tablet_bootstrap.cc:654] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.176395 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c2801d0bf442519ab3841a98d37696" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:53.176743 28434 raft_consensus.cc:2454] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:53.179177 28239 leader_election.cc:302] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:53.179857 352 raft_consensus.cc:2781] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:53.179946 352 raft_consensus.cc:482] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:53.180054 352 raft_consensus.cc:3037] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.182399 381 tablet_bootstrap.cc:492] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:53.182569 381 ts_tablet_manager.cc:1367] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240503 14:13:53.182881 389 ts_tablet_manager.cc:1398] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:13:53.183018 385 tablet_bootstrap.cc:492] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:53.183362 381 raft_consensus.cc:348] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.183583 381 raft_consensus.cc:374] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.183663 381 raft_consensus.cc:729] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.183921 381 consensus_queue.cc:260] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.184448 381 ts_tablet_manager.cc:1398] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:53.192446 352 raft_consensus.cc:504] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.193327 352 leader_election.cc:288] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:53.192668 385 tablet_bootstrap.cc:654] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.193642 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c2801d0bf442519ab3841a98d37696" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:53.193864 30939 raft_consensus.cc:3037] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.195112 386 tablet_bootstrap.cc:654] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.196695 30939 raft_consensus.cc:2454] T 16c2801d0bf442519ab3841a98d37696 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:53.196763 385 tablet_bootstrap.cc:492] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:53.196930 385 ts_tablet_manager.cc:1367] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240503 14:13:53.197609 385 raft_consensus.cc:348] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.197841 385 raft_consensus.cc:374] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.197928 385 raft_consensus.cc:729] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.198179 385 consensus_queue.cc:260] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.198804 385 ts_tablet_manager.cc:1398] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:53.199070 385 tablet_bootstrap.cc:492] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:53.200254 385 tablet_bootstrap.cc:654] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.201195 28239 leader_election.cc:302] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:53.202061 386 tablet_bootstrap.cc:492] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:53.202214 386 ts_tablet_manager.cc:1367] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.002s
I20240503 14:13:53.202839 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c2801d0bf442519ab3841a98d37696" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:53.203025 28434 raft_consensus.cc:3037] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.203066 386 raft_consensus.cc:348] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.203317 386 raft_consensus.cc:374] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.203404 386 raft_consensus.cc:729] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.203680 386 consensus_queue.cc:260] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.204347 386 ts_tablet_manager.cc:1398] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:53.205690 28434 raft_consensus.cc:2454] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:53.207156 352 raft_consensus.cc:2781] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:53.212415 385 tablet_bootstrap.cc:492] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:53.212677 385 ts_tablet_manager.cc:1367] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240503 14:13:53.213770 385 raft_consensus.cc:348] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.214162 385 raft_consensus.cc:374] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.214351 385 raft_consensus.cc:729] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.214665 385 consensus_queue.cc:260] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.215276 385 ts_tablet_manager.cc:1398] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:53.227600 403 raft_consensus.cc:482] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:53.227905 403 raft_consensus.cc:504] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.228977 403 leader_election.cc:288] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:53.229241 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c487a3f48f849298b3aec3155a559e4" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:53.229478 31202 raft_consensus.cc:2454] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:53.229700 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c487a3f48f849298b3aec3155a559e4" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:53.229923 28434 raft_consensus.cc:2454] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 0.
I20240503 14:13:53.229975 28238 leader_election.cc:302] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:53.230221 403 raft_consensus.cc:2781] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:53.230370 403 raft_consensus.cc:482] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:53.230468 403 raft_consensus.cc:3037] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.232908 403 raft_consensus.cc:504] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.233453 403 leader_election.cc:288] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:53.233911 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c487a3f48f849298b3aec3155a559e4" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:53.233891 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c487a3f48f849298b3aec3155a559e4" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:53.234112 28434 raft_consensus.cc:3037] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.234122 31202 raft_consensus.cc:3037] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.234964 352 raft_consensus.cc:686] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:53.239430 31202 raft_consensus.cc:2454] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:53.239444 28434 raft_consensus.cc:2454] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cdf05256a0407c8929ac43eb55d44f in term 1.
I20240503 14:13:53.240291 352 consensus_queue.cc:237] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.241220 28240 leader_election.cc:302] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:53.242020 352 raft_consensus.cc:2781] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:53.242147 352 raft_consensus.cc:686] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Leader. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:13:53.242432 352 consensus_queue.cc:237] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:53.245118 27894 catalog_manager.cc:5484] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:53.250451 27894 catalog_manager.cc:5484] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67). New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:53.322463 366 raft_consensus.cc:482] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:53.322680 366 raft_consensus.cc:504] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.323604 366 leader_election.cc:288] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:53.323952 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cbcf7c01bb4265a7c8fd4c897c4928" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:53.323949 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cbcf7c01bb4265a7c8fd4c897c4928" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:53.324172 28434 raft_consensus.cc:2454] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:53.324187 31070 raft_consensus.cc:2454] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:53.324618 30745 leader_election.cc:302] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:53.324947 366 raft_consensus.cc:2781] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:53.325071 366 raft_consensus.cc:482] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:53.325166 366 raft_consensus.cc:3037] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.327450 366 raft_consensus.cc:504] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.328014 366 leader_election.cc:288] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:53.328517 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cbcf7c01bb4265a7c8fd4c897c4928" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:53.328703 28432 raft_consensus.cc:3037] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.337014 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6cbcf7c01bb4265a7c8fd4c897c4928" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:53.337255 31070 raft_consensus.cc:3037] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.339591 31070 raft_consensus.cc:2454] T f6cbcf7c01bb4265a7c8fd4c897c4928 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:53.340091 28432 raft_consensus.cc:2454] T f6cbcf7c01bb4265a7c8fd4c897c4928 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:53.340655 30745 leader_election.cc:302] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:53.341416 32767 raft_consensus.cc:2781] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:53.341547 32767 raft_consensus.cc:686] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:53.341791 32767 consensus_queue.cc:237] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:53.345242 27896 catalog_manager.cc:5484] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:53.377590 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:53.393803 28432 raft_consensus.cc:1261] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.393870 31202 raft_consensus.cc:1261] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.395349 404 consensus_queue.cc:1035] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:53.397367 352 consensus_queue.cc:1035] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:53.410985 27894 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 7c487a3f48f849298b3aec3155a559e4 (table rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1]). Expected version 1 got 0
I20240503 14:13:53.412760 27894 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 7c487a3f48f849298b3aec3155a559e4 (table rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1]). Expected version 1 got 0
I20240503 14:13:53.424669 415 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:53.426399 414 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:53.426774 413 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:53.427093 414 tablet.cc:1620] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:53.428026 415 tablet.cc:1620] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:53.428431 413 tablet.cc:1620] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:53.428932 27894 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 7c487a3f48f849298b3aec3155a559e4 (table rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1]). Expected version 1 got 0
I20240503 14:13:53.429507 27896 catalog_manager.cc:5576] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331) does not have the latest schema for tablet 7c487a3f48f849298b3aec3155a559e4 (table rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1]). Expected version 1 got 0
I20240503 14:13:53.440917 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1] alter complete (version 1)
I20240503 14:13:53.535650 316 raft_consensus.cc:482] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:53.535836 316 raft_consensus.cc:504] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.536499 316 leader_election.cc:288] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:53.536938 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac53dcb19f941c48dd6c6cb623ccf58" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:53.536960 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac53dcb19f941c48dd6c6cb623ccf58" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:53.537155 31202 raft_consensus.cc:2454] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:53.537178 28043 raft_consensus.cc:2454] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:53.537609 28110 leader_election.cc:302] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:53.537901 363 raft_consensus.cc:2781] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:53.538017 363 raft_consensus.cc:482] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:53.538125 363 raft_consensus.cc:3037] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.540608 363 raft_consensus.cc:504] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.541227 363 leader_election.cc:288] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:53.541594 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac53dcb19f941c48dd6c6cb623ccf58" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:53.541659 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ac53dcb19f941c48dd6c6cb623ccf58" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:53.541867 31202 raft_consensus.cc:3037] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.541867 28043 raft_consensus.cc:3037] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.544715 28043 raft_consensus.cc:2454] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:53.544747 31202 raft_consensus.cc:2454] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:53.545075 28108 leader_election.cc:302] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:53.545305 363 raft_consensus.cc:2781] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:53.545464 363 raft_consensus.cc:686] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:53.545708 363 consensus_queue.cc:237] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:53.549257 27898 catalog_manager.cc:5484] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:53.552188 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:53.559427 413 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:53.559900 413 tablet.cc:1620] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:53.560101 414 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:53.561342 414 tablet.cc:1620] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: MemRowSet was empty: no flush needed.
I20240503 14:13:53.562470 415 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:53.562984 415 tablet.cc:1620] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:53.571725 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=97d85a8e56b7474fb045515d4108efb1] alter complete (version 2)
I20240503 14:13:53.645299 352 consensus_queue.cc:1035] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:53.651504 352 consensus_queue.cc:1035] T 16c2801d0bf442519ab3841a98d37696 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:53.674512 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:53.683251 413 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:53.684404 415 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:53.690191 414 tablet.cc:1722] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:53.696023 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=97d85a8e56b7474fb045515d4108efb1] alter complete (version 3)
I20240503 14:13:53.792043 32767 consensus_queue.cc:1035] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:53.797180 32767 consensus_queue.cc:1035] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:53.801326 27899 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:53.801575 27899 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:53.804852 27899 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 7c487a3f48f849298b3aec3155a559e4
I20240503 14:13:53.805547 31182 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c487a3f48f849298b3aec3155a559e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:53 UTC) from {username='slave'} at 127.0.0.1:38660
I20240503 14:13:53.806113 426 tablet_replica.cc:330] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:13:53.806182 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c487a3f48f849298b3aec3155a559e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:53 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:53.806334 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c487a3f48f849298b3aec3155a559e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:53 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:53.806382 426 raft_consensus.cc:2229] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:53.806596 426 raft_consensus.cc:2258] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:53.807610 426 ts_tablet_manager.cc:1875] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:53.815426 426 ts_tablet_manager.cc:1888] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:53.815585 426 log.cc:1196] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/7c487a3f48f849298b3aec3155a559e4
I20240503 14:13:53.816212 426 ts_tablet_manager.cc:1909] T 7c487a3f48f849298b3aec3155a559e4 P 530893a961af4b95b6b687fa5ed22f9d: Deleting consensus metadata
I20240503 14:13:53.817521 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 7c487a3f48f849298b3aec3155a559e4 (table rebalancer_dynamic_table_new_name [id=97d85a8e56b7474fb045515d4108efb1]) successfully deleted
I20240503 14:13:53.818624 427 tablet_replica.cc:330] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:53.818852 427 raft_consensus.cc:2229] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:53.819180 427 raft_consensus.cc:2258] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:53.819888 427 ts_tablet_manager.cc:1875] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:53.821710 428 tablet_replica.cc:330] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:53.821928 428 raft_consensus.cc:2229] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:53.822132 428 raft_consensus.cc:2258] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:53.823665 428 ts_tablet_manager.cc:1875] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:53.826231 427 ts_tablet_manager.cc:1888] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:53.826429 427 log.cc:1196] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/7c487a3f48f849298b3aec3155a559e4
I20240503 14:13:53.827008 427 ts_tablet_manager.cc:1909] T 7c487a3f48f849298b3aec3155a559e4 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:53.828925 428 ts_tablet_manager.cc:1888] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:53.829066 428 log.cc:1196] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c487a3f48f849298b3aec3155a559e4
I20240503 14:13:53.829190 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 7c487a3f48f849298b3aec3155a559e4 (table rebalancer_dynamic_table_new_name [id=97d85a8e56b7474fb045515d4108efb1]) successfully deleted
I20240503 14:13:53.829607 428 ts_tablet_manager.cc:1909] T 7c487a3f48f849298b3aec3155a559e4 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:53.830541 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 7c487a3f48f849298b3aec3155a559e4 (table rebalancer_dynamic_table_new_name [id=97d85a8e56b7474fb045515d4108efb1]) successfully deleted
I20240503 14:13:53.906957 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:53.914139 30921 tablet_service.cc:1467] Processing CreateTablet for tablet fdcafcb5beff4026be9794ffee2530a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d]), partition=
I20240503 14:13:53.914588 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcafcb5beff4026be9794ffee2530a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.914593 31051 tablet_service.cc:1467] Processing CreateTablet for tablet fdcafcb5beff4026be9794ffee2530a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d]), partition=
I20240503 14:13:53.914891 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcafcb5beff4026be9794ffee2530a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.918592 28154 tablet_service.cc:1467] Processing CreateTablet for tablet fdcafcb5beff4026be9794ffee2530a9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d]), partition=
I20240503 14:13:53.919006 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcafcb5beff4026be9794ffee2530a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.919507 429 tablet_bootstrap.cc:492] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:53.921414 429 tablet_bootstrap.cc:654] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.921777 430 tablet_bootstrap.cc:492] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:53.923925 430 tablet_bootstrap.cc:654] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.926292 430 tablet_bootstrap.cc:492] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:53.926446 430 ts_tablet_manager.cc:1367] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 14:13:53.927230 430 raft_consensus.cc:348] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.927479 430 raft_consensus.cc:374] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.927598 430 raft_consensus.cc:729] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.927894 430 consensus_queue.cc:260] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.930810 430 ts_tablet_manager.cc:1398] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:53.931983 433 tablet_bootstrap.cc:492] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:53.932377 429 tablet_bootstrap.cc:492] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:53.932533 429 ts_tablet_manager.cc:1367] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240503 14:13:53.933331 429 raft_consensus.cc:348] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.933578 429 raft_consensus.cc:374] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.933738 429 raft_consensus.cc:729] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.934131 429 consensus_queue.cc:260] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.934775 429 ts_tablet_manager.cc:1398] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:53.934381 433 tablet_bootstrap.cc:654] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:53.941723 433 tablet_bootstrap.cc:492] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:53.941872 433 ts_tablet_manager.cc:1367] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:13:53.942917 433 raft_consensus.cc:348] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.943223 433 raft_consensus.cc:374] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:53.943360 433 raft_consensus.cc:729] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:53.943698 433 consensus_queue.cc:260] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.945686 433 ts_tablet_manager.cc:1398] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:53.980865 363 raft_consensus.cc:482] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:53.981020 363 raft_consensus.cc:504] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.981606 363 leader_election.cc:288] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:53.982595 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcafcb5beff4026be9794ffee2530a9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:53.982836 30940 raft_consensus.cc:2454] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:53.983358 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcafcb5beff4026be9794ffee2530a9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:53.983618 31072 raft_consensus.cc:2454] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:53.984161 28109 leader_election.cc:302] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:53.984475 316 raft_consensus.cc:2781] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:53.984591 316 raft_consensus.cc:482] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:53.984683 316 raft_consensus.cc:3037] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.986797 316 raft_consensus.cc:504] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.987290 316 leader_election.cc:288] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:53.987679 30940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcafcb5beff4026be9794ffee2530a9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:53.987694 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcafcb5beff4026be9794ffee2530a9" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:53.987846 30940 raft_consensus.cc:3037] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.987923 31072 raft_consensus.cc:3037] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:53.989665 30940 raft_consensus.cc:2454] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:53.990025 28109 leader_election.cc:302] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:53.990178 31072 raft_consensus.cc:2454] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:53.990890 363 raft_consensus.cc:2781] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:53.991086 363 raft_consensus.cc:686] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:53.991326 363 consensus_queue.cc:237] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:53.995522 27898 catalog_manager.cc:5484] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:54.003683 363 consensus_queue.cc:1035] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.016036 316 consensus_queue.cc:1035] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.063408 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:54.073269 30790 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2df0906f1e49e2a36b4e4ad1dbc418 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.073743 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2df0906f1e49e2a36b4e4ad1dbc418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.074195 28023 tablet_service.cc:1467] Processing CreateTablet for tablet da7bb0a1956a452cbc1cbbb3999fd7e2 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.074566 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7bb0a1956a452cbc1cbbb3999fd7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.074615 28154 tablet_service.cc:1467] Processing CreateTablet for tablet da7bb0a1956a452cbc1cbbb3999fd7e2 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.074645 31182 tablet_service.cc:1467] Processing CreateTablet for tablet da7bb0a1956a452cbc1cbbb3999fd7e2 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.075008 31182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7bb0a1956a452cbc1cbbb3999fd7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.075028 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7bb0a1956a452cbc1cbbb3999fd7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.078517 28021 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2df0906f1e49e2a36b4e4ad1dbc418 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.078827 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 8c2df0906f1e49e2a36b4e4ad1dbc418 (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.079205 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2df0906f1e49e2a36b4e4ad1dbc418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.081537 31051 tablet_service.cc:1467] Processing CreateTablet for tablet 68ab254ac309475984065af1a01b537f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.081950 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ab254ac309475984065af1a01b537f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.082654 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 68ab254ac309475984065af1a01b537f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.083055 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ab254ac309475984065af1a01b537f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.083139 31183 tablet_service.cc:1467] Processing CreateTablet for tablet 68ab254ac309475984065af1a01b537f (DEFAULT_TABLE table=rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.083514 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ab254ac309475984065af1a01b537f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.090480 430 tablet_bootstrap.cc:492] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:54.092016 430 tablet_bootstrap.cc:654] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.095500 441 tablet_bootstrap.cc:492] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:54.097299 441 tablet_bootstrap.cc:654] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.099682 429 tablet_bootstrap.cc:492] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:54.101128 442 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:54.101611 429 tablet_bootstrap.cc:654] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.103442 442 tablet_bootstrap.cc:654] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.103507 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:54.108168 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c2df0906f1e49e2a36b4e4ad1dbc418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.109505 433 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:54.111284 433 tablet_bootstrap.cc:654] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.112793 447 tablet_bootstrap.cc:492] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:54.114722 433 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:54.114795 447 tablet_bootstrap.cc:654] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.114884 433 ts_tablet_manager.cc:1367] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 14:13:54.115368 430 tablet_bootstrap.cc:492] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:54.115537 430 ts_tablet_manager.cc:1367] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.007s
I20240503 14:13:54.115643 433 raft_consensus.cc:348] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.115885 433 raft_consensus.cc:374] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.115955 433 raft_consensus.cc:729] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.116317 430 raft_consensus.cc:348] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.116412 433 consensus_queue.cc:260] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.116581 430 raft_consensus.cc:374] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.116654 430 raft_consensus.cc:729] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.116890 430 consensus_queue.cc:260] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.117926 430 ts_tablet_manager.cc:1398] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:54.119720 30940 raft_consensus.cc:1261] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.119881 433 ts_tablet_manager.cc:1398] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:54.120947 442 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:54.121089 442 ts_tablet_manager.cc:1367] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240503 14:13:54.121834 442 raft_consensus.cc:348] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.122035 442 raft_consensus.cc:374] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.122126 442 raft_consensus.cc:729] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.122402 442 consensus_queue.cc:260] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.122701 437 consensus_queue.cc:1035] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.122941 442 ts_tablet_manager.cc:1398] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:54.123248 442 tablet_bootstrap.cc:492] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:54.129079 441 tablet_bootstrap.cc:492] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:54.129369 441 ts_tablet_manager.cc:1367] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.008s
I20240503 14:13:54.130438 31070 raft_consensus.cc:1261] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.130348 441 raft_consensus.cc:348] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.130693 441 raft_consensus.cc:374] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.130851 441 raft_consensus.cc:729] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.131131 437 consensus_queue.cc:1035] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.131146 441 consensus_queue.cc:260] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.131846 441 ts_tablet_manager.cc:1398] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:54.134595 437 raft_consensus.cc:482] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:54.134733 437 raft_consensus.cc:504] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.135414 437 leader_election.cc:288] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:54.137135 442 tablet_bootstrap.cc:654] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.137739 451 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:54.138401 27901 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet fdcafcb5beff4026be9794ffee2530a9 (table rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d]). Expected version 1 got 0
I20240503 14:13:54.139699 451 tablet_bootstrap.cc:654] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.144047 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bb0a1956a452cbc1cbbb3999fd7e2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:54.144320 31202 raft_consensus.cc:2454] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 0.
I20240503 14:13:54.145872 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bb0a1956a452cbc1cbbb3999fd7e2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:54.146142 442 tablet_bootstrap.cc:492] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:54.146394 442 ts_tablet_manager.cc:1367] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.007s
I20240503 14:13:54.147292 442 raft_consensus.cc:348] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.147725 442 raft_consensus.cc:374] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.147889 442 raft_consensus.cc:729] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.148200 442 consensus_queue.cc:260] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.148985 442 ts_tablet_manager.cc:1398] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20240503 14:13:54.149345 28110 leader_election.cc:341] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:54.149776 28108 leader_election.cc:302] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:13:54.157166 451 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:54.157402 451 ts_tablet_manager.cc:1367] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.004s
I20240503 14:13:54.158290 451 raft_consensus.cc:348] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.158583 451 raft_consensus.cc:374] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.158700 451 raft_consensus.cc:729] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.158967 451 consensus_queue.cc:260] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.159584 451 ts_tablet_manager.cc:1398] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:54.173399 316 raft_consensus.cc:2781] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:54.173568 316 raft_consensus.cc:482] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:54.173667 316 raft_consensus.cc:3037] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.181037 453 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:54.181712 453 tablet.cc:1620] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:54.203827 451 tablet_bootstrap.cc:492] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:54.205340 451 tablet_bootstrap.cc:654] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.206687 316 raft_consensus.cc:504] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.208175 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bb0a1956a452cbc1cbbb3999fd7e2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:54.208406 28043 raft_consensus.cc:3037] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.208621 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d] alter complete (version 1)
I20240503 14:13:54.211273 447 tablet_bootstrap.cc:492] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:54.211417 447 ts_tablet_manager.cc:1367] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.006s
I20240503 14:13:54.211683 28043 raft_consensus.cc:2454] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:54.212289 447 raft_consensus.cc:348] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.212575 447 raft_consensus.cc:374] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.212713 447 raft_consensus.cc:729] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.213016 447 consensus_queue.cc:260] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.213757 447 ts_tablet_manager.cc:1398] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:54.214558 28110 leader_election.cc:302] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:54.214905 457 raft_consensus.cc:2781] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:54.215166 457 raft_consensus.cc:686] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:13:54.215819 457 consensus_queue.cc:237] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:54.223282 27899 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet fdcafcb5beff4026be9794ffee2530a9 (table rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d]). Expected version 1 got 0
I20240503 14:13:54.223789 451 tablet_bootstrap.cc:492] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:54.223963 451 ts_tablet_manager.cc:1367] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20240503 14:13:54.224632 451 raft_consensus.cc:348] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.224900 451 raft_consensus.cc:374] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.224993 451 raft_consensus.cc:729] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.225247 451 consensus_queue.cc:260] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.225768 451 ts_tablet_manager.cc:1398] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:54.244010 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da7bb0a1956a452cbc1cbbb3999fd7e2" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:54.244236 31202 raft_consensus.cc:3037] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.250970 31202 raft_consensus.cc:2454] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:13:54.252848 316 leader_election.cc:288] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 1 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:54.257134 455 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:54.258558 417 raft_consensus.cc:482] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:54.258744 417 raft_consensus.cc:504] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.258963 455 tablet.cc:1620] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:54.259449 417 leader_election.cc:288] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:54.260649 429 tablet_bootstrap.cc:492] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:54.260684 27899 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet fdcafcb5beff4026be9794ffee2530a9 (table rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d]). Expected version 1 got 0
I20240503 14:13:54.260845 429 ts_tablet_manager.cc:1367] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.161s user 0.011s sys 0.000s
I20240503 14:13:54.262029 429 raft_consensus.cc:348] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.262259 429 raft_consensus.cc:374] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.262369 429 raft_consensus.cc:729] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.262687 429 consensus_queue.cc:260] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.263147 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ab254ac309475984065af1a01b537f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:54.263379 31070 raft_consensus.cc:2454] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 0.
I20240503 14:13:54.263522 429 ts_tablet_manager.cc:1398] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:54.263947 31138 leader_election.cc:302] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:54.264277 417 raft_consensus.cc:2781] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:54.264394 417 raft_consensus.cc:482] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:54.264483 417 raft_consensus.cc:3037] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.267213 417 raft_consensus.cc:504] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.267848 417 leader_election.cc:288] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:54.268343 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ab254ac309475984065af1a01b537f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:54.268544 30941 raft_consensus.cc:3037] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.270838 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ab254ac309475984065af1a01b537f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:54.271072 31070 raft_consensus.cc:3037] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.273530 31070 raft_consensus.cc:2454] T 68ab254ac309475984065af1a01b537f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:54.274353 31138 leader_election.cc:302] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:54.275272 32610 raft_consensus.cc:2781] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:54.275434 32610 raft_consensus.cc:686] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:13:54.275683 32610 consensus_queue.cc:237] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:54.276355 30941 raft_consensus.cc:2454] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:13:54.277118 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ab254ac309475984065af1a01b537f" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:54.277292 30939 raft_consensus.cc:2362] T 68ab254ac309475984065af1a01b537f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1. Re-sending same reply.
I20240503 14:13:54.282313 27894 catalog_manager.cc:5484] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:54.283711 27899 catalog_manager.cc:5484] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:54.291990 461 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:54.292578 461 tablet.cc:1620] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:54.295328 27899 catalog_manager.cc:5576] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707) does not have the latest schema for tablet fdcafcb5beff4026be9794ffee2530a9 (table rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d]). Expected version 1 got 0
I20240503 14:13:54.320989 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:54.328461 453 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:54.328964 453 tablet.cc:1620] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:54.329118 461 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:54.329586 461 tablet.cc:1620] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: MemRowSet was empty: no flush needed.
I20240503 14:13:54.329993 455 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:54.330487 455 tablet.cc:1620] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:54.343338 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=93b5f45fabe54f2c9d42785a94abdb4d] alter complete (version 2)
I20240503 14:13:54.380825 32767 raft_consensus.cc:482] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:54.381018 32767 raft_consensus.cc:504] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.381703 32767 leader_election.cc:288] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:54.382128 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2df0906f1e49e2a36b4e4ad1dbc418" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:54.382258 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2df0906f1e49e2a36b4e4ad1dbc418" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:13:54.382344 28304 raft_consensus.cc:2454] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:54.382488 28043 raft_consensus.cc:2454] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:54.382889 30746 leader_election.cc:302] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:54.383157 32767 raft_consensus.cc:2781] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:54.383255 32767 raft_consensus.cc:482] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:54.383355 32767 raft_consensus.cc:3037] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.386337 32767 raft_consensus.cc:504] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.387027 32767 leader_election.cc:288] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:13:54.387488 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2df0906f1e49e2a36b4e4ad1dbc418" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:13:54.387516 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c2df0906f1e49e2a36b4e4ad1dbc418" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:54.387668 28043 raft_consensus.cc:3037] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.387728 28304 raft_consensus.cc:3037] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.390434 28304 raft_consensus.cc:2454] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:54.390799 30746 leader_election.cc:302] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:54.391201 32767 raft_consensus.cc:2781] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:54.390338 28043 raft_consensus.cc:2454] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:54.393667 32767 raft_consensus.cc:686] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:54.393934 32767 consensus_queue.cc:237] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:54.399039 27899 catalog_manager.cc:5484] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:54.446957 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:54.455050 453 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:54.455789 455 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:54.457661 461 tablet.cc:1722] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:54.465415 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=93b5f45fabe54f2c9d42785a94abdb4d] alter complete (version 3)
I20240503 14:13:54.569475 27898 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:54.569725 27898 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:54.572300 27898 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet fdcafcb5beff4026be9794ffee2530a9
I20240503 14:13:54.572999 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcafcb5beff4026be9794ffee2530a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:54 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:54.573280 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcafcb5beff4026be9794ffee2530a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:54 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:54.573866 467 tablet_replica.cc:330] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:54.573906 31051 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcafcb5beff4026be9794ffee2530a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:54 UTC) from {username='slave'} at 127.0.0.1:45512
I20240503 14:13:54.574101 467 raft_consensus.cc:2229] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:54.574559 467 raft_consensus.cc:2258] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:54.575335 467 ts_tablet_manager.cc:1875] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:54.579890 467 ts_tablet_manager.cc:1888] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:54.580000 467 log.cc:1196] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/fdcafcb5beff4026be9794ffee2530a9
I20240503 14:13:54.580449 467 ts_tablet_manager.cc:1909] T fdcafcb5beff4026be9794ffee2530a9 P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:54.581555 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet fdcafcb5beff4026be9794ffee2530a9 (table rebalancer_dynamic_table_new_name [id=93b5f45fabe54f2c9d42785a94abdb4d]) successfully deleted
I20240503 14:13:54.582594 468 tablet_replica.cc:330] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:13:54.582793 468 raft_consensus.cc:2229] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:54.582974 468 raft_consensus.cc:2258] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:54.583070 466 tablet_replica.cc:330] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:54.583287 466 raft_consensus.cc:2229] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:54.583530 466 raft_consensus.cc:2258] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:54.583880 468 ts_tablet_manager.cc:1875] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:54.584385 466 ts_tablet_manager.cc:1875] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:54.590111 468 ts_tablet_manager.cc:1888] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:54.590310 468 log.cc:1196] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/fdcafcb5beff4026be9794ffee2530a9
I20240503 14:13:54.590821 468 ts_tablet_manager.cc:1909] T fdcafcb5beff4026be9794ffee2530a9 P fde134d0fa2b4900b9badd2e08fea09d: Deleting consensus metadata
I20240503 14:13:54.591979 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet fdcafcb5beff4026be9794ffee2530a9 (table rebalancer_dynamic_table_new_name [id=93b5f45fabe54f2c9d42785a94abdb4d]) successfully deleted
I20240503 14:13:54.592312 466 ts_tablet_manager.cc:1888] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:54.592443 466 log.cc:1196] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/fdcafcb5beff4026be9794ffee2530a9
I20240503 14:13:54.592948 466 ts_tablet_manager.cc:1909] T fdcafcb5beff4026be9794ffee2530a9 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:54.593815 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet fdcafcb5beff4026be9794ffee2530a9 (table rebalancer_dynamic_table_new_name [id=93b5f45fabe54f2c9d42785a94abdb4d]) successfully deleted
I20240503 14:13:54.674877 27898 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:54.682003 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 583dcaf1beb14474b88041abea0eca95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9]), partition=
I20240503 14:13:54.682442 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583dcaf1beb14474b88041abea0eca95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.682670 30921 tablet_service.cc:1467] Processing CreateTablet for tablet 583dcaf1beb14474b88041abea0eca95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9]), partition=
I20240503 14:13:54.682997 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583dcaf1beb14474b88041abea0eca95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.686826 28283 tablet_service.cc:1467] Processing CreateTablet for tablet 583dcaf1beb14474b88041abea0eca95 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9]), partition=
I20240503 14:13:54.687124 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 583dcaf1beb14474b88041abea0eca95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.687662 469 tablet_bootstrap.cc:492] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:54.689119 469 tablet_bootstrap.cc:654] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.691517 469 tablet_bootstrap.cc:492] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:54.691661 469 ts_tablet_manager.cc:1367] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:54.692648 471 tablet_bootstrap.cc:492] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:54.692533 469 raft_consensus.cc:348] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.692836 469 raft_consensus.cc:374] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.692981 469 raft_consensus.cc:729] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.693038 457 consensus_queue.cc:1035] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.693485 469 consensus_queue.cc:260] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.693915 429 tablet_bootstrap.cc:492] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:54.694314 469 ts_tablet_manager.cc:1398] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:54.694471 471 tablet_bootstrap.cc:654] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.695425 429 tablet_bootstrap.cc:654] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.701097 471 tablet_bootstrap.cc:492] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:54.701292 471 ts_tablet_manager.cc:1367] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20240503 14:13:54.702399 471 raft_consensus.cc:348] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.703104 471 raft_consensus.cc:374] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.703238 471 raft_consensus.cc:729] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.703531 471 consensus_queue.cc:260] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.704170 471 ts_tablet_manager.cc:1398] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:54.714332 429 tablet_bootstrap.cc:492] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:54.714536 429 ts_tablet_manager.cc:1367] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240503 14:13:54.715512 429 raft_consensus.cc:348] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.715788 429 raft_consensus.cc:374] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:54.715915 429 raft_consensus.cc:729] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:54.716199 429 consensus_queue.cc:260] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.717007 457 consensus_queue.cc:1035] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.722769 429 ts_tablet_manager.cc:1398] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:13:54.732735 32635 raft_consensus.cc:482] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:54.732918 32635 raft_consensus.cc:504] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.733541 32635 leader_election.cc:288] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:54.734002 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583dcaf1beb14474b88041abea0eca95" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:54.734014 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583dcaf1beb14474b88041abea0eca95" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:54.734313 28302 raft_consensus.cc:2454] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:54.734316 28432 raft_consensus.cc:2454] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:54.734760 30877 leader_election.cc:302] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:13:54.735040 32635 raft_consensus.cc:2781] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:54.735136 32635 raft_consensus.cc:482] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:54.735231 32635 raft_consensus.cc:3037] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.737716 32635 raft_consensus.cc:504] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.738729 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583dcaf1beb14474b88041abea0eca95" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:54.738940 28302 raft_consensus.cc:3037] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.740835 32635 leader_election.cc:288] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:54.741758 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583dcaf1beb14474b88041abea0eca95" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:54.741986 28432 raft_consensus.cc:3037] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:54.744259 28432 raft_consensus.cc:2454] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:54.744602 30877 leader_election.cc:302] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d; no voters:
I20240503 14:13:54.744877 32635 raft_consensus.cc:2781] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:54.745008 32635 raft_consensus.cc:686] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:54.745062 28302 raft_consensus.cc:2454] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:54.745265 32635 consensus_queue.cc:237] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:54.748311 27898 catalog_manager.cc:5484] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:54.773633 32610 consensus_queue.cc:1035] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.779788 32610 consensus_queue.cc:1035] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.861825 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:54.868233 28303 raft_consensus.cc:1261] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.868822 28432 raft_consensus.cc:1261] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.869467 475 consensus_queue.cc:1035] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.869791 32636 consensus_queue.cc:1035] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.897632 455 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:54.898156 455 tablet.cc:1620] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:54.900578 27899 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 583dcaf1beb14474b88041abea0eca95 (table rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9]). Expected version 1 got 0
I20240503 14:13:54.913308 27898 catalog_manager.cc:5576] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889) does not have the latest schema for tablet 583dcaf1beb14474b88041abea0eca95 (table rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9]). Expected version 1 got 0
I20240503 14:13:54.914810 488 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:54.915414 488 tablet.cc:1620] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:54.915530 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9] alter complete (version 1)
I20240503 14:13:54.924109 487 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:54.924714 487 tablet.cc:1620] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:54.928658 27894 catalog_manager.cc:5576] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371) does not have the latest schema for tablet 583dcaf1beb14474b88041abea0eca95 (table rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9]). Expected version 1 got 0
I20240503 14:13:54.931157 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:54.939940 31051 tablet_service.cc:1467] Processing CreateTablet for tablet d4c83d087d9c457db809bb3fabecf260 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.940347 31051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c83d087d9c457db809bb3fabecf260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.940321 31183 tablet_service.cc:1467] Processing CreateTablet for tablet d4c83d087d9c457db809bb3fabecf260 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.940752 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c83d087d9c457db809bb3fabecf260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.941321 30790 tablet_service.cc:1467] Processing CreateTablet for tablet f4b9d916f3fb4a348a8702662c143435 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.941720 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b9d916f3fb4a348a8702662c143435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.942660 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 9a49047eedc54168af70a7b59fd7d770 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.943053 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a49047eedc54168af70a7b59fd7d770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.943861 31052 tablet_service.cc:1467] Processing CreateTablet for tablet 9a49047eedc54168af70a7b59fd7d770 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.944231 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a49047eedc54168af70a7b59fd7d770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.943893 31050 tablet_service.cc:1467] Processing CreateTablet for tablet f4b9d916f3fb4a348a8702662c143435 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.944705 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b9d916f3fb4a348a8702662c143435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.947211 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 9a49047eedc54168af70a7b59fd7d770 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.947710 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a49047eedc54168af70a7b59fd7d770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.949263 28282 tablet_service.cc:1467] Processing CreateTablet for tablet f4b9d916f3fb4a348a8702662c143435 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.949666 28282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b9d916f3fb4a348a8702662c143435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.949265 28283 tablet_service.cc:1467] Processing CreateTablet for tablet d4c83d087d9c457db809bb3fabecf260 (DEFAULT_TABLE table=rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:54.950280 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4c83d087d9c457db809bb3fabecf260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.960691 469 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:54.962435 469 tablet_bootstrap.cc:654] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.964515 464 consensus_queue.cc:1035] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.972860 495 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:54.975205 495 tablet_bootstrap.cc:654] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.976068 471 tablet_bootstrap.cc:492] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:54.977983 471 tablet_bootstrap.cc:654] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.985579 493 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:54.988338 493 tablet_bootstrap.cc:654] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:54.990566 464 consensus_queue.cc:1035] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:54.992348 491 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:54.994511 491 tablet_bootstrap.cc:654] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.004678 495 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:55.004907 495 ts_tablet_manager.cc:1367] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.007s
I20240503 14:13:55.005789 495 raft_consensus.cc:348] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.006012 471 tablet_bootstrap.cc:492] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:55.006083 495 raft_consensus.cc:374] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.006244 495 raft_consensus.cc:729] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.006287 471 ts_tablet_manager.cc:1367] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.007s
I20240503 14:13:55.006583 495 consensus_queue.cc:260] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.007200 471 raft_consensus.cc:348] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.007738 471 raft_consensus.cc:374] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.007875 471 raft_consensus.cc:729] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.009341 495 ts_tablet_manager.cc:1398] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:55.009678 495 tablet_bootstrap.cc:492] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:55.011464 495 tablet_bootstrap.cc:654] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.013458 471 consensus_queue.cc:260] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.019641 493 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:55.019869 493 ts_tablet_manager.cc:1367] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20240503 14:13:55.018616 27898 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:55.020756 493 raft_consensus.cc:348] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.021095 493 raft_consensus.cc:374] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.021252 493 raft_consensus.cc:729] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.021596 493 consensus_queue.cc:260] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.022416 493 ts_tablet_manager.cc:1398] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:55.022677 491 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:55.022882 491 ts_tablet_manager.cc:1367] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.006s
I20240503 14:13:55.024569 491 raft_consensus.cc:348] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.024883 491 raft_consensus.cc:374] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.025038 491 raft_consensus.cc:729] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.025368 491 consensus_queue.cc:260] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.026196 491 ts_tablet_manager.cc:1398] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:55.031194 464 raft_consensus.cc:482] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:55.031360 464 raft_consensus.cc:504] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.032117 464 leader_election.cc:288] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:55.034406 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b9d916f3fb4a348a8702662c143435" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:55.034972 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b9d916f3fb4a348a8702662c143435" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:55.035054 455 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:55.035257 31070 raft_consensus.cc:2454] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 0.
I20240503 14:13:55.035619 455 tablet.cc:1620] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:55.037559 30745 leader_election.cc:302] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
W20240503 14:13:55.038975 30746 leader_election.cc:341] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:55.039786 32767 raft_consensus.cc:2781] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:55.039883 32767 raft_consensus.cc:482] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:55.039981 32767 raft_consensus.cc:3037] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.041754 487 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:55.042407 487 tablet.cc:1620] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:55.044734 488 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:55.045533 488 tablet.cc:1620] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: MemRowSet was empty: no flush needed.
I20240503 14:13:55.057721 27896 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=f2fdf942bdb94369bf96124f90d768e9] alter complete (version 2)
I20240503 14:13:55.066002 32767 raft_consensus.cc:504] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.073241 32767 leader_election.cc:288] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:55.073866 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b9d916f3fb4a348a8702662c143435" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
W20240503 14:13:55.074393 30746 leader_election.cc:341] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:55.075527 495 tablet_bootstrap.cc:492] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:55.075903 495 ts_tablet_manager.cc:1367] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.066s user 0.027s sys 0.015s
I20240503 14:13:55.076977 495 raft_consensus.cc:348] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.077318 495 raft_consensus.cc:374] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.077453 495 raft_consensus.cc:729] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.077821 495 consensus_queue.cc:260] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.078596 495 ts_tablet_manager.cc:1398] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:55.079088 495 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:55.081205 495 tablet_bootstrap.cc:654] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.090418 471 ts_tablet_manager.cc:1398] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.084s user 0.002s sys 0.000s
I20240503 14:13:55.104501 471 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:55.107710 495 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:55.107960 495 ts_tablet_manager.cc:1367] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.029s user 0.016s sys 0.008s
I20240503 14:13:55.109114 495 raft_consensus.cc:348] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.109454 495 raft_consensus.cc:374] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.109596 495 raft_consensus.cc:729] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.109915 495 consensus_queue.cc:260] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.110200 471 tablet_bootstrap.cc:654] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.112777 495 ts_tablet_manager.cc:1398] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:55.112931 469 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:55.113207 469 ts_tablet_manager.cc:1367] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.153s user 0.008s sys 0.025s
I20240503 14:13:55.114295 469 raft_consensus.cc:348] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.114697 469 raft_consensus.cc:374] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.114836 469 raft_consensus.cc:729] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.115247 469 consensus_queue.cc:260] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.120652 469 ts_tablet_manager.cc:1398] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
W20240503 14:13:55.124770 30745 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 85154.4us
I20240503 14:13:55.124960 489 tablet_bootstrap.cc:492] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:55.125525 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b9d916f3fb4a348a8702662c143435" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:55.125720 31072 raft_consensus.cc:3037] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.126569 489 tablet_bootstrap.cc:654] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.128667 471 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:55.129236 31072 raft_consensus.cc:2454] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:13:55.128829 471 ts_tablet_manager.cc:1367] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240503 14:13:55.130507 471 raft_consensus.cc:348] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.130713 471 raft_consensus.cc:374] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.130798 471 raft_consensus.cc:729] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.131042 471 consensus_queue.cc:260] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.133194 30745 leader_election.cc:302] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:13:55.133440 464 raft_consensus.cc:2781] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:55.133595 464 raft_consensus.cc:686] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:13:55.133611 471 ts_tablet_manager.cc:1398] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:55.133852 464 consensus_queue.cc:237] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.135121 489 tablet_bootstrap.cc:492] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:55.135495 489 ts_tablet_manager.cc:1367] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240503 14:13:55.136227 489 raft_consensus.cc:348] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.136536 489 raft_consensus.cc:374] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.136662 489 raft_consensus.cc:729] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.136989 489 consensus_queue.cc:260] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.137748 489 ts_tablet_manager.cc:1398] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:55.149900 27899 catalog_manager.cc:5484] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:55.173192 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:55.176472 477 raft_consensus.cc:482] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:55.176643 477 raft_consensus.cc:504] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.177343 477 leader_election.cc:288] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:55.179899 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c83d087d9c457db809bb3fabecf260" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:55.180094 28303 raft_consensus.cc:2454] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:55.180519 31008 leader_election.cc:302] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:55.181528 455 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:55.182010 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c83d087d9c457db809bb3fabecf260" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:55.182528 376 raft_consensus.cc:2781] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:55.182622 376 raft_consensus.cc:482] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:55.182221 31202 raft_consensus.cc:2454] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:55.182704 376 raft_consensus.cc:3037] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.185835 376 raft_consensus.cc:504] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.186480 376 leader_election.cc:288] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:13:55.186676 487 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:55.188038 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c83d087d9c457db809bb3fabecf260" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:55.188230 28304 raft_consensus.cc:3037] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.189436 488 tablet.cc:1722] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:55.192874 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4c83d087d9c457db809bb3fabecf260" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:55.193084 31202 raft_consensus.cc:3037] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.194598 28304 raft_consensus.cc:2454] T d4c83d087d9c457db809bb3fabecf260 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:55.195401 27899 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=f2fdf942bdb94369bf96124f90d768e9] alter complete (version 3)
I20240503 14:13:55.196573 31008 leader_election.cc:302] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:55.196947 477 raft_consensus.cc:2781] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:55.197108 477 raft_consensus.cc:686] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:55.197378 477 consensus_queue.cc:237] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:13:55.202572 31202 raft_consensus.cc:2454] T d4c83d087d9c457db809bb3fabecf260 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:55.204025 27894 catalog_manager.cc:5484] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:13:55.286700 32230 raft_consensus.cc:482] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:55.286839 32230 raft_consensus.cc:504] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.287529 32230 leader_election.cc:288] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:55.287994 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a49047eedc54168af70a7b59fd7d770" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:55.288110 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a49047eedc54168af70a7b59fd7d770" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:55.288264 28173 raft_consensus.cc:2454] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:55.288317 31070 raft_consensus.cc:2454] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:55.288691 28369 leader_election.cc:302] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:55.288949 32230 raft_consensus.cc:2781] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:55.289057 32230 raft_consensus.cc:482] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:55.289142 32230 raft_consensus.cc:3037] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.291561 32230 raft_consensus.cc:504] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.292111 32230 leader_election.cc:288] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:13:55.292426 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a49047eedc54168af70a7b59fd7d770" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:55.292642 28173 raft_consensus.cc:3037] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.292718 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a49047eedc54168af70a7b59fd7d770" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:13:55.292927 31070 raft_consensus.cc:3037] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.295550 28173 raft_consensus.cc:2454] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:55.295655 31070 raft_consensus.cc:2454] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:55.296003 28368 leader_election.cc:302] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:55.296262 32230 raft_consensus.cc:2781] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:55.296363 32230 raft_consensus.cc:686] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:55.296535 32230 consensus_queue.cc:237] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:13:55.299229 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:55.299415 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:55.302858 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet 583dcaf1beb14474b88041abea0eca95
I20240503 14:13:55.303593 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 583dcaf1beb14474b88041abea0eca95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:55 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:55.304102 510 tablet_replica.cc:330] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:55.304088 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 583dcaf1beb14474b88041abea0eca95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:55 UTC) from {username='slave'} at 127.0.0.1:43626
I20240503 14:13:55.304236 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 583dcaf1beb14474b88041abea0eca95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:55 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:55.304363 510 raft_consensus.cc:2229] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.304536 510 raft_consensus.cc:2258] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.305068 512 tablet_replica.cc:330] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:55.305419 512 raft_consensus.cc:2229] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:55.305591 510 ts_tablet_manager.cc:1875] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:55.305905 512 raft_consensus.cc:2258] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.307583 512 ts_tablet_manager.cc:1875] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:55.310441 27899 catalog_manager.cc:5484] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:13:55.314344 512 ts_tablet_manager.cc:1888] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:55.314532 512 log.cc:1196] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/583dcaf1beb14474b88041abea0eca95
I20240503 14:13:55.314538 510 ts_tablet_manager.cc:1888] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:55.314790 510 log.cc:1196] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/583dcaf1beb14474b88041abea0eca95
I20240503 14:13:55.315172 512 ts_tablet_manager.cc:1909] T 583dcaf1beb14474b88041abea0eca95 P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:55.315297 510 ts_tablet_manager.cc:1909] T 583dcaf1beb14474b88041abea0eca95 P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:55.316252 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 583dcaf1beb14474b88041abea0eca95 (table rebalancer_dynamic_table_new_name [id=f2fdf942bdb94369bf96124f90d768e9]) successfully deleted
I20240503 14:13:55.316890 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 583dcaf1beb14474b88041abea0eca95 (table rebalancer_dynamic_table_new_name [id=f2fdf942bdb94369bf96124f90d768e9]) successfully deleted
I20240503 14:13:55.317646 511 tablet_replica.cc:330] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:13:55.317842 511 raft_consensus.cc:2229] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.318030 511 raft_consensus.cc:2258] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.318763 511 ts_tablet_manager.cc:1875] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:55.324097 511 ts_tablet_manager.cc:1888] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:55.324201 511 log.cc:1196] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/583dcaf1beb14474b88041abea0eca95
I20240503 14:13:55.324579 511 ts_tablet_manager.cc:1909] T 583dcaf1beb14474b88041abea0eca95 P a4cdf05256a0407c8929ac43eb55d44f: Deleting consensus metadata
I20240503 14:13:55.325469 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 583dcaf1beb14474b88041abea0eca95 (table rebalancer_dynamic_table_new_name [id=f2fdf942bdb94369bf96124f90d768e9]) successfully deleted
I20240503 14:13:55.392604 398 raft_consensus.cc:482] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:55.392804 398 raft_consensus.cc:504] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.393502 398 leader_election.cc:288] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:55.393889 31072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b9d916f3fb4a348a8702662c143435" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:13:55.393913 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b9d916f3fb4a348a8702662c143435" candidate_uuid: "a4cdf05256a0407c8929ac43eb55d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:55.394181 31072 raft_consensus.cc:2379] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a4cdf05256a0407c8929ac43eb55d44f in current term 1: Already voted for candidate d0dbfb0e43f948c3922926aacc365fc1 in this term.
I20240503 14:13:55.394552 28239 leader_election.cc:302] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4cdf05256a0407c8929ac43eb55d44f; no voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:13:55.394799 398 raft_consensus.cc:3037] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.397476 398 raft_consensus.cc:2726] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:55.410666 27899 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:55.418629 28412 tablet_service.cc:1467] Processing CreateTablet for tablet a392cee9b17a4766b220b07e65ef3e6f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9]), partition=
I20240503 14:13:55.418648 30921 tablet_service.cc:1467] Processing CreateTablet for tablet a392cee9b17a4766b220b07e65ef3e6f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9]), partition=
I20240503 14:13:55.418984 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a392cee9b17a4766b220b07e65ef3e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.419018 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a392cee9b17a4766b220b07e65ef3e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.424158 469 tablet_bootstrap.cc:492] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:55.424487 28154 tablet_service.cc:1467] Processing CreateTablet for tablet a392cee9b17a4766b220b07e65ef3e6f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9]), partition=
I20240503 14:13:55.424597 513 tablet_bootstrap.cc:492] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:55.424875 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a392cee9b17a4766b220b07e65ef3e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.425284 469 tablet_bootstrap.cc:654] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.427961 513 tablet_bootstrap.cc:654] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.428637 469 tablet_bootstrap.cc:492] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:55.428767 469 ts_tablet_manager.cc:1367] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:55.429505 469 raft_consensus.cc:348] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.429701 469 raft_consensus.cc:374] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.429755 469 raft_consensus.cc:729] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.429976 469 consensus_queue.cc:260] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.432615 469 ts_tablet_manager.cc:1398] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:55.439901 491 tablet_bootstrap.cc:492] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:55.440979 513 tablet_bootstrap.cc:492] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:55.441090 513 ts_tablet_manager.cc:1367] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20240503 14:13:55.441227 491 tablet_bootstrap.cc:654] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.441704 513 raft_consensus.cc:348] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.441898 513 raft_consensus.cc:374] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.441984 513 raft_consensus.cc:729] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.442181 513 consensus_queue.cc:260] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.443122 491 tablet_bootstrap.cc:492] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:55.443264 491 ts_tablet_manager.cc:1367] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:55.443881 491 raft_consensus.cc:348] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.444082 491 raft_consensus.cc:374] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.444147 491 raft_consensus.cc:729] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.444183 513 ts_tablet_manager.cc:1398] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:55.444352 491 consensus_queue.cc:260] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.445554 491 ts_tablet_manager.cc:1398] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:55.458608 32230 raft_consensus.cc:482] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:55.458751 32230 raft_consensus.cc:504] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.459364 32230 leader_election.cc:288] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:55.460018 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a392cee9b17a4766b220b07e65ef3e6f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:55.460206 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a392cee9b17a4766b220b07e65ef3e6f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:55.460268 30939 raft_consensus.cc:2454] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:55.460403 28173 raft_consensus.cc:2454] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 0.
I20240503 14:13:55.460713 28368 leader_election.cc:302] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:55.460961 32230 raft_consensus.cc:2781] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:55.461040 32230 raft_consensus.cc:482] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:55.461118 32230 raft_consensus.cc:3037] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.463176 32230 raft_consensus.cc:504] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.463760 32230 leader_election.cc:288] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:55.464540 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a392cee9b17a4766b220b07e65ef3e6f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:55.464730 30939 raft_consensus.cc:3037] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.464639 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a392cee9b17a4766b220b07e65ef3e6f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:55.465335 28173 raft_consensus.cc:3037] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:55.468537 28173 raft_consensus.cc:2454] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:55.468993 28368 leader_election.cc:302] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:13:55.469244 32230 raft_consensus.cc:2781] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:55.469470 32230 raft_consensus.cc:686] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:13:55.470093 30939 raft_consensus.cc:2454] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:13:55.470045 32230 consensus_queue.cc:237] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:55.474180 27894 catalog_manager.cc:5484] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:55.588368 27894 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:55.594504 30939 raft_consensus.cc:1261] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:55.594884 28173 raft_consensus.cc:1261] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:55.595415 32595 consensus_queue.cc:1035] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.595605 32230 consensus_queue.cc:1035] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.600698 487 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:55.601115 487 tablet.cc:1620] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:55.601408 455 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:55.601855 455 tablet.cc:1620] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:55.602838 27899 catalog_manager.cc:5576] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465) does not have the latest schema for tablet a392cee9b17a4766b220b07e65ef3e6f (table rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9]). Expected version 1 got 0
I20240503 14:13:55.603624 27898 catalog_manager.cc:5576] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039) does not have the latest schema for tablet a392cee9b17a4766b220b07e65ef3e6f (table rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9]). Expected version 1 got 0
I20240503 14:13:55.611267 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9] alter complete (version 1)
I20240503 14:13:55.615773 523 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:55.617005 523 tablet.cc:1620] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:55.620411 27899 catalog_manager.cc:5576] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039) does not have the latest schema for tablet a392cee9b17a4766b220b07e65ef3e6f (table rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9]). Expected version 1 got 0
I20240503 14:13:55.623395 522 consensus_queue.cc:1035] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.654997 522 consensus_queue.cc:1035] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.661813 32767 consensus_queue.cc:1035] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.665045 32767 consensus_queue.cc:1035] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.715113 27899 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:55.721729 487 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:55.722254 487 tablet.cc:1620] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: MemRowSet was empty: no flush needed.
I20240503 14:13:55.723646 523 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:55.724074 523 tablet.cc:1620] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: MemRowSet was empty: no flush needed.
I20240503 14:13:55.726701 455 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:55.727097 455 tablet.cc:1620] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: MemRowSet was empty: no flush needed.
I20240503 14:13:55.732883 27894 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table [id=b03754dbb5ca42a19ac29c35f4d252b9] alter complete (version 2)
I20240503 14:13:55.756891 521 consensus_queue.cc:1035] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.761888 521 consensus_queue.cc:1035] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:55.838042 27896 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:55.848107 487 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:55.848963 455 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:55.853680 523 tablet.cc:1722] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:55.866310 27898 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: rebalancer_dynamic_table_new_name [id=b03754dbb5ca42a19ac29c35f4d252b9] alter complete (version 3)
I20240503 14:13:55.886135 27896 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_extra_table_66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:55.904223 30790 tablet_service.cc:1467] Processing CreateTablet for tablet ee914d743a2c46418f79728f3d32d1c5 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.904708 30790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee914d743a2c46418f79728f3d32d1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.904954 28412 tablet_service.cc:1467] Processing CreateTablet for tablet ee914d743a2c46418f79728f3d32d1c5 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.905328 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee914d743a2c46418f79728f3d32d1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.905421 30921 tablet_service.cc:1467] Processing CreateTablet for tablet ee914d743a2c46418f79728f3d32d1c5 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.905828 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee914d743a2c46418f79728f3d32d1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.913023 513 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:55.914672 513 tablet_bootstrap.cc:654] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.914569 28021 tablet_service.cc:1467] Processing CreateTablet for tablet e1e4fa6d5c704d7f9bd43767b4f77e5b (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.914995 28021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e4fa6d5c704d7f9bd43767b4f77e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.923704 513 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:55.923929 513 ts_tablet_manager.cc:1367] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240503 14:13:55.924857 513 raft_consensus.cc:348] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:55.925695 513 raft_consensus.cc:374] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.925846 513 raft_consensus.cc:729] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.926151 513 consensus_queue.cc:260] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:55.927239 513 ts_tablet_manager.cc:1398] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:55.931512 31183 tablet_service.cc:1467] Processing CreateTablet for tablet e1e4fa6d5c704d7f9bd43767b4f77e5b (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.931844 28283 tablet_service.cc:1467] Processing CreateTablet for tablet e1e4fa6d5c704d7f9bd43767b4f77e5b (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.931980 31183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e4fa6d5c704d7f9bd43767b4f77e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.932243 28283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e4fa6d5c704d7f9bd43767b4f77e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.932343 28023 tablet_service.cc:1467] Processing CreateTablet for tablet f7bdba20d1844eefb6d3b29c5deb2873 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.932718 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7bdba20d1844eefb6d3b29c5deb2873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.934727 469 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:55.935662 30789 tablet_service.cc:1467] Processing CreateTablet for tablet f7bdba20d1844eefb6d3b29c5deb2873 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.936080 30789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7bdba20d1844eefb6d3b29c5deb2873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.936153 469 tablet_bootstrap.cc:654] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.939042 30921 tablet_service.cc:1467] Processing CreateTablet for tablet f7bdba20d1844eefb6d3b29c5deb2873 (DEFAULT_TABLE table=rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:55.939558 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7bdba20d1844eefb6d3b29c5deb2873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.947302 513 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:55.947820 469 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:55.948017 469 ts_tablet_manager.cc:1367] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20240503 14:13:55.948175 536 tablet_bootstrap.cc:492] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:13:55.948901 469 raft_consensus.cc:348] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:55.949123 469 raft_consensus.cc:374] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.949246 469 raft_consensus.cc:729] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.949519 469 consensus_queue.cc:260] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:55.950714 469 ts_tablet_manager.cc:1398] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:55.951781 536 tablet_bootstrap.cc:654] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.952980 513 tablet_bootstrap.cc:654] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.953070 537 tablet_bootstrap.cc:492] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:55.968955 27894 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:55.969152 27894 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33234:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:55.971521 536 tablet_bootstrap.cc:492] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d: No bootstrap required, opened a new log
I20240503 14:13:55.971784 536 ts_tablet_manager.cc:1367] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.003s
I20240503 14:13:55.972604 536 raft_consensus.cc:348] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.973170 536 raft_consensus.cc:374] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.973323 536 raft_consensus.cc:729] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.973659 536 consensus_queue.cc:260] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.974341 27894 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 5baa17487b4b4e1eb60d2f5fcb4de355: Sending DeleteTablet for 3 replicas of tablet a392cee9b17a4766b220b07e65ef3e6f
I20240503 14:13:55.974642 536 ts_tablet_manager.cc:1398] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:55.976444 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet a392cee9b17a4766b220b07e65ef3e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:55 UTC) from {username='slave'} at 127.0.0.1:54338
I20240503 14:13:55.976994 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet a392cee9b17a4766b220b07e65ef3e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:55 UTC) from {username='slave'} at 127.0.0.1:58086
I20240503 14:13:55.977216 541 tablet_replica.cc:330] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:13:55.977553 541 raft_consensus.cc:2229] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.977648 542 tablet_replica.cc:330] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:13:55.977917 542 raft_consensus.cc:2229] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:55.977914 541 raft_consensus.cc:2258] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.978332 542 raft_consensus.cc:2258] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.979286 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet a392cee9b17a4766b220b07e65ef3e6f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:55 UTC) from {username='slave'} at 127.0.0.1:35238
I20240503 14:13:55.980255 542 ts_tablet_manager.cc:1875] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:55.981843 543 tablet_replica.cc:330] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:13:55.982187 543 raft_consensus.cc:2229] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.982507 543 raft_consensus.cc:2258] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.984001 537 tablet_bootstrap.cc:654] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.984109 541 ts_tablet_manager.cc:1875] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:55.984745 543 ts_tablet_manager.cc:1875] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:55.985016 534 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:55.988951 513 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d: No bootstrap required, opened a new log
I20240503 14:13:55.989044 534 tablet_bootstrap.cc:654] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.989619 513 ts_tablet_manager.cc:1367] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.004s
I20240503 14:13:55.990628 513 raft_consensus.cc:348] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:55.990859 513 raft_consensus.cc:374] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.990981 513 raft_consensus.cc:729] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.991250 513 consensus_queue.cc:260] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:55.994027 537 tablet_bootstrap.cc:492] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:55.994264 537 ts_tablet_manager.cc:1367] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.002s
I20240503 14:13:55.994755 540 tablet_bootstrap.cc:492] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:13:55.995815 537 raft_consensus.cc:348] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:55.996161 537 raft_consensus.cc:374] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:55.996284 537 raft_consensus.cc:729] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:55.996588 540 tablet_bootstrap.cc:654] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:55.996634 537 consensus_queue.cc:260] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:56.000620 542 ts_tablet_manager.cc:1888] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:56.002568 542 log.cc:1196] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/a392cee9b17a4766b220b07e65ef3e6f
I20240503 14:13:56.002810 537 ts_tablet_manager.cc:1398] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240503 14:13:56.003090 542 ts_tablet_manager.cc:1909] T a392cee9b17a4766b220b07e65ef3e6f P 5388902973134958a030fd63ab4faba4: Deleting consensus metadata
I20240503 14:13:56.004496 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet a392cee9b17a4766b220b07e65ef3e6f (table rebalancer_dynamic_table_new_name [id=b03754dbb5ca42a19ac29c35f4d252b9]) successfully deleted
I20240503 14:13:56.006464 513 ts_tablet_manager.cc:1398] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:56.012279 543 ts_tablet_manager.cc:1888] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:56.013013 421 raft_consensus.cc:482] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:56.013159 421 raft_consensus.cc:504] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:56.013906 421 leader_election.cc:288] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:56.015005 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4fa6d5c704d7f9bd43767b4f77e5b" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:13:56.015242 31203 raft_consensus.cc:2454] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:56.015815 27977 leader_election.cc:302] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:56.016245 540 tablet_bootstrap.cc:492] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f: No bootstrap required, opened a new log
I20240503 14:13:56.016480 540 ts_tablet_manager.cc:1367] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.003s
I20240503 14:13:56.016799 421 raft_consensus.cc:2781] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:56.016894 421 raft_consensus.cc:482] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:56.016963 421 raft_consensus.cc:3037] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.017483 540 raft_consensus.cc:348] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:56.017864 540 raft_consensus.cc:374] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:56.018010 540 raft_consensus.cc:729] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:13:56.018285 540 consensus_queue.cc:260] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:56.018959 540 ts_tablet_manager.cc:1398] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:56.019309 543 log.cc:1196] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/a392cee9b17a4766b220b07e65ef3e6f
I20240503 14:13:56.019690 541 ts_tablet_manager.cc:1888] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:56.019783 543 ts_tablet_manager.cc:1909] T a392cee9b17a4766b220b07e65ef3e6f P 6409961ee8c0445687c11724213c44a9: Deleting consensus metadata
I20240503 14:13:56.021482 421 raft_consensus.cc:504] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:56.022068 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4fa6d5c704d7f9bd43767b4f77e5b" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f" is_pre_election: true
I20240503 14:13:56.022305 28302 raft_consensus.cc:2454] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:56.022359 541 log.cc:1196] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/a392cee9b17a4766b220b07e65ef3e6f
I20240503 14:13:56.022845 541 ts_tablet_manager.cc:1909] T a392cee9b17a4766b220b07e65ef3e6f P 6bbc686598f24f518d7ca6bb2063058d: Deleting consensus metadata
I20240503 14:13:56.023692 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4fa6d5c704d7f9bd43767b4f77e5b" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:13:56.023932 31202 raft_consensus.cc:3037] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.027757 31202 raft_consensus.cc:2454] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:56.028211 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet a392cee9b17a4766b220b07e65ef3e6f (table rebalancer_dynamic_table_new_name [id=b03754dbb5ca42a19ac29c35f4d252b9]) successfully deleted
I20240503 14:13:56.028663 27977 leader_election.cc:302] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:56.028921 463 raft_consensus.cc:2781] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:56.029199 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet a392cee9b17a4766b220b07e65ef3e6f (table rebalancer_dynamic_table_new_name [id=b03754dbb5ca42a19ac29c35f4d252b9]) successfully deleted
I20240503 14:13:56.029441 421 leader_election.cc:288] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:13:56.029937 537 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:56.030122 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e4fa6d5c704d7f9bd43767b4f77e5b" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:13:56.030341 28302 raft_consensus.cc:3037] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.031358 537 tablet_bootstrap.cc:654] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:56.033888 28302 raft_consensus.cc:2454] T e1e4fa6d5c704d7f9bd43767b4f77e5b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:56.047381 534 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:56.049043 534 ts_tablet_manager.cc:1367] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.000s
I20240503 14:13:56.051615 534 raft_consensus.cc:348] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:56.054535 534 raft_consensus.cc:374] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:56.054647 534 raft_consensus.cc:729] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:56.055181 534 consensus_queue.cc:260] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:56.055745 534 ts_tablet_manager.cc:1398] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:56.059888 534 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:13:56.065523 534 tablet_bootstrap.cc:654] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:56.078492 27894 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33234:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:56.094190 463 raft_consensus.cc:686] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:56.095304 537 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: No bootstrap required, opened a new log
I20240503 14:13:56.095575 537 ts_tablet_manager.cc:1367] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.066s user 0.020s sys 0.012s
I20240503 14:13:56.095532 463 consensus_queue.cc:237] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:13:56.096609 537 raft_consensus.cc:348] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:56.096943 537 raft_consensus.cc:374] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:56.097339 537 raft_consensus.cc:729] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:56.098168 537 consensus_queue.cc:260] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:56.098851 537 ts_tablet_manager.cc:1398] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:56.106037 27894 catalog_manager.cc:5484] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:56.124557 28412 tablet_service.cc:1467] Processing CreateTablet for tablet 9517b56a435b48e18c79d2a375e5a951 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=168110ceedbf4e228913b8743651cb17]), partition=
I20240503 14:13:56.125023 28412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9517b56a435b48e18c79d2a375e5a951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:56.127625 28154 tablet_service.cc:1467] Processing CreateTablet for tablet 9517b56a435b48e18c79d2a375e5a951 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=168110ceedbf4e228913b8743651cb17]), partition=
I20240503 14:13:56.128022 28154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9517b56a435b48e18c79d2a375e5a951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:56.128543 31050 tablet_service.cc:1467] Processing CreateTablet for tablet 9517b56a435b48e18c79d2a375e5a951 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=168110ceedbf4e228913b8743651cb17]), partition=
I20240503 14:13:56.128847 31050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9517b56a435b48e18c79d2a375e5a951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:56.137437 469 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:13:56.138909 469 tablet_bootstrap.cc:654] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:56.142284 534 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1: No bootstrap required, opened a new log
I20240503 14:13:56.142627 534 ts_tablet_manager.cc:1367] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.083s user 0.010s sys 0.008s
I20240503 14:13:56.143628 534 raft_consensus.cc:348] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:56.144865 534 raft_consensus.cc:374] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:56.145130 534 raft_consensus.cc:729] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:13:56.145166 553 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:13:56.145628 534 consensus_queue.cc:260] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:56.148643 553 tablet_bootstrap.cc:654] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:56.149442 534 ts_tablet_manager.cc:1398] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:56.160773 555 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:56.162155 555 tablet_bootstrap.cc:654] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:56.162658 469 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4: No bootstrap required, opened a new log
I20240503 14:13:56.162847 469 ts_tablet_manager.cc:1367] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.009s
I20240503 14:13:56.164286 469 raft_consensus.cc:348] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.164631 469 raft_consensus.cc:374] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:56.164693 469 raft_consensus.cc:729] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:13:56.164887 469 consensus_queue.cc:260] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.165477 469 ts_tablet_manager.cc:1398] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:56.177428 553 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: No bootstrap required, opened a new log
I20240503 14:13:56.177656 553 ts_tablet_manager.cc:1367] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.006s
I20240503 14:13:56.178314 553 raft_consensus.cc:348] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.178547 553 raft_consensus.cc:374] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:56.178653 553 raft_consensus.cc:729] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:13:56.178936 553 consensus_queue.cc:260] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.183425 555 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9: No bootstrap required, opened a new log
I20240503 14:13:56.183588 555 ts_tablet_manager.cc:1367] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.005s
I20240503 14:13:56.184409 555 raft_consensus.cc:348] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.184635 555 raft_consensus.cc:374] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:56.184692 555 raft_consensus.cc:729] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:56.184916 555 consensus_queue.cc:260] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.185263 553 ts_tablet_manager.cc:1398] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:56.185504 555 ts_tablet_manager.cc:1398] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:56.196434 32635 raft_consensus.cc:482] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:56.196630 32635 raft_consensus.cc:504] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:56.197587 32635 leader_election.cc:288] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:56.197996 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee914d743a2c46418f79728f3d32d1c5" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:56.198308 30809 raft_consensus.cc:2454] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:56.198601 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee914d743a2c46418f79728f3d32d1c5" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:56.198846 28433 raft_consensus.cc:2454] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 0.
I20240503 14:13:56.198940 30874 leader_election.cc:302] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:56.199676 32635 raft_consensus.cc:2781] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:56.199807 32635 raft_consensus.cc:482] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:56.199895 32635 raft_consensus.cc:3037] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.203341 32635 raft_consensus.cc:504] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:56.204100 32635 leader_election.cc:288] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:56.204560 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee914d743a2c46418f79728f3d32d1c5" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:56.204746 28433 raft_consensus.cc:3037] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.209712 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee914d743a2c46418f79728f3d32d1c5" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:56.209937 30809 raft_consensus.cc:3037] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.212077 30809 raft_consensus.cc:2454] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:56.212477 30874 leader_election.cc:302] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:13:56.212852 32635 raft_consensus.cc:2781] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:56.213029 32635 raft_consensus.cc:686] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:13:56.213357 32635 consensus_queue.cc:237] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:13:56.215101 28433 raft_consensus.cc:2454] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 1.
I20240503 14:13:56.219561 27896 catalog_manager.cc:5484] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:13:56.233444 550 raft_consensus.cc:482] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:56.233598 550 raft_consensus.cc:504] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:56.234350 550 leader_election.cc:288] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:56.234809 30938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d" is_pre_election: true
I20240503 14:13:56.235050 30938 raft_consensus.cc:2454] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:56.235108 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:13:56.235323 30809 raft_consensus.cc:2454] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 0.
I20240503 14:13:56.235494 27978 leader_election.cc:302] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:56.235834 550 raft_consensus.cc:2781] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:56.235913 550 raft_consensus.cc:482] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:56.235975 550 raft_consensus.cc:3037] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.238947 550 raft_consensus.cc:504] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:56.240243 550 leader_election.cc:288] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:13:56.240223 30938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:13:56.240406 30938 raft_consensus.cc:3037] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.242758 30938 raft_consensus.cc:2454] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:56.243115 27978 leader_election.cc:302] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:13:56.243783 549 raft_consensus.cc:2781] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:56.243897 549 raft_consensus.cc:686] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:13:56.243942 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:13:56.244125 30809 raft_consensus.cc:3037] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.244174 549 consensus_queue.cc:237] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:56.252171 30809 raft_consensus.cc:2454] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:13:56.258231 27899 catalog_manager.cc:5484] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 0 to 1, leader changed from <none> to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:56.295527 376 raft_consensus.cc:482] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:56.296116 376 raft_consensus.cc:504] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.296996 376 leader_election.cc:288] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:56.297391 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:13:56.297609 28433 raft_consensus.cc:2454] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:56.297607 28171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:13:56.297807 28171 raft_consensus.cc:2454] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 0.
I20240503 14:13:56.298031 31008 leader_election.cc:302] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:56.298434 376 raft_consensus.cc:2781] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:56.298516 376 raft_consensus.cc:482] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:56.298621 376 raft_consensus.cc:3037] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.300901 376 raft_consensus.cc:504] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.302047 376 leader_election.cc:288] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:56.302495 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:13:56.302536 28171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:13:56.302709 28433 raft_consensus.cc:3037] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.302708 28171 raft_consensus.cc:3037] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:56.304678 28171 raft_consensus.cc:2454] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:56.305162 31006 leader_election.cc:302] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:13:56.305434 376 raft_consensus.cc:2781] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:56.305931 376 raft_consensus.cc:686] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:13:56.306100 28433 raft_consensus.cc:2454] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:13:56.306959 376 consensus_queue.cc:237] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:56.320659 27899 catalog_manager.cc:5484] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 0 to 1, leader changed from <none> to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:56.494347 549 consensus_queue.cc:1035] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:56.521131 547 consensus_queue.cc:1035] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:56.697826 32635 consensus_queue.cc:1035] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:56.709056 32636 consensus_queue.cc:1035] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:56.719645 376 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:56.729825 376 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:56.834985 547 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:56.841313 547 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:57.293023 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:13:57.299824 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:13:57.329569 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:13:57.345741 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:57.358527 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:57.363952 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:13:57.405998 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:13:57.426934 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:13:57.972203 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:13:57.977471 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:13:58.026624 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:58.034508 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:13:58.051833 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:13:58.079335 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:13:58.102877 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:58.119665 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:13:58.695607 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:13:58.711270 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:13:58.741508 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:58.776010 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:13:58.828104 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:13:58.854704 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:13:58.898443 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:58.920805 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:13:59.232825 28172 consensus_queue.cc:237] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
W20240503 14:13:59.235327 28109 consensus_peers.cc:485] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45da3dab7a204d378273a8985d91ace8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.235347 31203 raft_consensus.cc:1261] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.235881 30809 raft_consensus.cc:1261] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.236008 761 consensus_queue.cc:1035] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.236362 802 consensus_queue.cc:1035] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.239893 761 raft_consensus.cc:2932] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.240434 30809 raft_consensus.cc:2932] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.244344 31203 raft_consensus.cc:2932] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.248373 27893 catalog_manager.cc:5484] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.298292 28432 consensus_queue.cc:237] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:59.300750 28172 raft_consensus.cc:1261] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.301508 31070 raft_consensus.cc:1261] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:59.301764 28370 consensus_peers.cc:485] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a49047eedc54168af70a7b59fd7d770. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.301985 674 consensus_queue.cc:1035] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.302498 674 consensus_queue.cc:1035] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.306838 826 raft_consensus.cc:2932] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.307312 31070 raft_consensus.cc:2932] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.312753 28172 raft_consensus.cc:2932] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.314139 27896 catalog_manager.cc:5484] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.364889 30939 consensus_queue.cc:237] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:59.375166 28302 raft_consensus.cc:1261] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:59.375536 30875 consensus_peers.cc:485] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6b4ef3ee1fd440384272cee496fc102. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.375734 32636 consensus_queue.cc:1035] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.377787 31203 raft_consensus.cc:1261] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.378382 32636 consensus_queue.cc:1035] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.381035 32635 raft_consensus.cc:2932] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.381508 31203 raft_consensus.cc:2932] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.387627 27898 catalog_manager.cc:5484] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.389834 28302 raft_consensus.cc:2932] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.427105 31070 consensus_queue.cc:237] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
W20240503 14:13:59.430318 31008 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3888675106d4d389718380627c95edf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.430759 28433 raft_consensus.cc:1261] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.430766 30939 raft_consensus.cc:1261] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.431273 764 consensus_queue.cc:1035] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.432027 764 consensus_queue.cc:1035] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.435662 522 raft_consensus.cc:2932] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:59.436205 28433 raft_consensus.cc:2932] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:59.441869 30939 raft_consensus.cc:2932] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:59.444309 27893 catalog_manager.cc:5484] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:13:59.491497 28040 consensus_queue.cc:237] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
W20240503 14:13:59.494431 27978 consensus_peers.cc:485] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 344eb942dc3d4188bc9baf3d1ecc4f7a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.494635 28302 raft_consensus.cc:1261] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.495484 622 consensus_queue.cc:1035] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.495784 30808 raft_consensus.cc:1261] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.496660 622 consensus_queue.cc:1035] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.511502 622 raft_consensus.cc:2932] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.512161 30808 raft_consensus.cc:2932] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.522861 27896 catalog_manager.cc:5484] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.528689 28302 raft_consensus.cc:2932] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.569012 31072 consensus_queue.cc:237] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:59.571923 28433 raft_consensus.cc:1261] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:59.572042 31008 consensus_peers.cc:485] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9517b56a435b48e18c79d2a375e5a951. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.572762 28171 raft_consensus.cc:1261] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.573312 522 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.573643 764 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.577450 764 raft_consensus.cc:2932] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.578203 28433 raft_consensus.cc:2932] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.584707 27898 catalog_manager.cc:5484] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.589242 28171 raft_consensus.cc:2932] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.641882 30939 consensus_queue.cc:237] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:59.661232 31203 raft_consensus.cc:1261] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:59.663697 30875 consensus_peers.cc:485] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 016a1c41112441619700c662beee8c98. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.664651 32636 consensus_queue.cc:1035] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.680756 28433 raft_consensus.cc:1261] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.681274 32635 consensus_queue.cc:1035] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.691264 924 ts_tablet_manager.cc:902] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:13:59.693980 923 raft_consensus.cc:2932] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.694458 31203 raft_consensus.cc:2932] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.699574 28433 raft_consensus.cc:2932] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.702538 27896 catalog_manager.cc:5484] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:13:59.722914 924 tablet_copy_client.cc:308] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:13:59.724193 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet 45da3dab7a204d378273a8985d91ace8 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:46183)
I20240503 14:13:59.724433 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet 45da3dab7a204d378273a8985d91ace8 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:46183: session id = 6bbc686598f24f518d7ca6bb2063058d-45da3dab7a204d378273a8985d91ace8
I20240503 14:13:59.727723 28194 tablet_copy_source_session.cc:205] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:59.730477 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45da3dab7a204d378273a8985d91ace8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:59.739329 924 tablet_copy_client.cc:783] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:59.739682 924 tablet_copy_client.cc:647] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:59.742741 924 tablet_copy_client.cc:523] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:59.746866 924 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:13:59.754050 924 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:59.754508 924 tablet_bootstrap.cc:492] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:13:59.754765 924 ts_tablet_manager.cc:1367] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20240503 14:13:59.755506 924 raft_consensus.cc:348] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:59.755707 924 raft_consensus.cc:729] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:13:59.756036 924 consensus_queue.cc:260] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:59.757217 924 ts_tablet_manager.cc:1398] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:59.758731 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-45da3dab7a204d378273a8985d91ace8 received from {username='slave'} at 127.20.240.70:46183
I20240503 14:13:59.758896 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-45da3dab7a204d378273a8985d91ace8 on tablet 45da3dab7a204d378273a8985d91ace8 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:13:59.772533 28302 consensus_queue.cc:237] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:13:59.775274 28433 raft_consensus.cc:1261] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:59.775936 784 consensus_queue.cc:1035] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:59.776136 28171 raft_consensus.cc:1261] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:59.775945 28237 consensus_peers.cc:485] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd84de9286eb4ad7b88b9f243eb2801c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.776829 784 consensus_queue.cc:1035] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:59.780400 784 raft_consensus.cc:2932] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:59.781117 28171 raft_consensus.cc:2932] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:59.785006 28433 raft_consensus.cc:2932] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:59.786329 27896 catalog_manager.cc:5484] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:13:59.826478 30808 consensus_queue.cc:237] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:13:59.829943 28302 raft_consensus.cc:1261] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:59.829962 30745 consensus_peers.cc:485] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4b9d916f3fb4a348a8702662c143435. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.830863 31072 raft_consensus.cc:1261] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.831488 868 consensus_queue.cc:1035] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.832723 969 ts_tablet_manager.cc:902] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:13:59.832718 568 consensus_queue.cc:1035] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:13:59.842496 969 tablet_copy_client.cc:308] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:13:59.843510 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet a6b4ef3ee1fd440384272cee496fc102 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:54479)
I20240503 14:13:59.843714 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet a6b4ef3ee1fd440384272cee496fc102 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:54479: session id = 6409961ee8c0445687c11724213c44a9-a6b4ef3ee1fd440384272cee496fc102
I20240503 14:13:59.846489 30961 tablet_copy_source_session.cc:205] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:59.848322 969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6b4ef3ee1fd440384272cee496fc102. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:59.850749 568 raft_consensus.cc:2932] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.851466 28303 raft_consensus.cc:2932] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.858325 31072 raft_consensus.cc:2932] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.859447 27898 catalog_manager.cc:5484] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:13:59.869414 969 tablet_copy_client.cc:783] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:59.869725 969 tablet_copy_client.cc:647] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:59.876267 976 ts_tablet_manager.cc:902] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:13:59.880203 969 tablet_copy_client.cc:523] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:59.884267 969 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:13:59.907409 969 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:59.907898 969 tablet_bootstrap.cc:492] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:13:59.908159 969 ts_tablet_manager.cc:1367] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.006s
I20240503 14:13:59.912479 969 raft_consensus.cc:348] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:59.912806 969 raft_consensus.cc:729] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:13:59.914219 969 consensus_queue.cc:260] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:13:59.915365 976 tablet_copy_client.cc:308] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:13:59.916576 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 9a49047eedc54168af70a7b59fd7d770 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:55553)
I20240503 14:13:59.916812 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 9a49047eedc54168af70a7b59fd7d770 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:55553: session id = 793a0d5dd1634c5faaf0e1e4851e8294-9a49047eedc54168af70a7b59fd7d770
I20240503 14:13:59.921923 28454 tablet_copy_source_session.cc:205] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:59.925428 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a49047eedc54168af70a7b59fd7d770. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:59.930187 969 ts_tablet_manager.cc:1398] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.022s user 0.001s sys 0.003s
I20240503 14:13:59.931474 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-a6b4ef3ee1fd440384272cee496fc102 received from {username='slave'} at 127.20.240.66:54479
I20240503 14:13:59.931665 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 6409961ee8c0445687c11724213c44a9-a6b4ef3ee1fd440384272cee496fc102 on tablet a6b4ef3ee1fd440384272cee496fc102 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:13:59.940389 976 tablet_copy_client.cc:783] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:59.940726 976 tablet_copy_client.cc:647] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:59.942945 976 tablet_copy_client.cc:523] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:59.945806 976 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:13:59.957350 976 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:59.957736 976 tablet_bootstrap.cc:492] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:13:59.957978 976 ts_tablet_manager.cc:1367] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240503 14:13:59.958686 976 raft_consensus.cc:348] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:59.958900 976 raft_consensus.cc:729] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:13:59.959154 976 consensus_queue.cc:260] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:13:59.961313 976 ts_tablet_manager.cc:1398] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:59.962013 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-9a49047eedc54168af70a7b59fd7d770 received from {username='slave'} at 127.20.240.65:55553
I20240503 14:13:59.962184 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-9a49047eedc54168af70a7b59fd7d770 on tablet 9a49047eedc54168af70a7b59fd7d770 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:13:59.979599 31072 consensus_queue.cc:237] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
W20240503 14:13:59.982995 31008 consensus_peers.cc:485] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 618ba33946424065a465217bae7cdbb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.983098 28171 raft_consensus.cc:1261] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.983350 31203 raft_consensus.cc:1261] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:59.983621 1000 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.984072 999 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:59.986950 999 raft_consensus.cc:2932] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.988123 31203 raft_consensus.cc:2932] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.993726 27896 catalog_manager.cc:5484] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:13:59.995432 28171 raft_consensus.cc:2932] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:00.022589 1013 ts_tablet_manager.cc:902] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:14:00.025220 1013 tablet_copy_client.cc:308] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:14:00.026432 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet a3888675106d4d389718380627c95edf from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:54103)
I20240503 14:14:00.026692 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet a3888675106d4d389718380627c95edf from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:54103: session id = a4cdf05256a0407c8929ac43eb55d44f-a3888675106d4d389718380627c95edf
I20240503 14:14:00.029995 31092 tablet_copy_source_session.cc:205] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.032183 30939 consensus_queue.cc:237] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:00.032459 1013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3888675106d4d389718380627c95edf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.035204 31072 raft_consensus.cc:1261] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:00.035319 31203 raft_consensus.cc:1261] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:00.035554 30875 consensus_peers.cc:485] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9132d0f9d1ea4be49458a085c9e98a65. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.036082 32636 consensus_queue.cc:1035] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:00.037688 32636 consensus_queue.cc:1035] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:00.042667 1013 tablet_copy_client.cc:783] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.042994 1013 tablet_copy_client.cc:647] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.045750 1013 tablet_copy_client.cc:523] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.048854 1013 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:00.053526 32635 raft_consensus.cc:2932] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.054279 31203 raft_consensus.cc:2932] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.060068 27898 catalog_manager.cc:5484] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.063645 31072 raft_consensus.cc:2932] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.065845 924 ts_tablet_manager.cc:902] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:00.104075 1013 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.104653 1013 tablet_bootstrap.cc:492] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:00.104940 1013 ts_tablet_manager.cc:1367] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.020s
I20240503 14:14:00.105897 1013 raft_consensus.cc:348] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:00.106262 1013 raft_consensus.cc:729] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.106546 1013 consensus_queue.cc:260] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:00.107544 1013 ts_tablet_manager.cc:1398] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240503 14:14:00.108542 924 tablet_copy_client.cc:308] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:00.108917 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-a3888675106d4d389718380627c95edf received from {username='slave'} at 127.20.240.67:54103
I20240503 14:14:00.109103 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-a3888675106d4d389718380627c95edf on tablet a3888675106d4d389718380627c95edf with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:00.110051 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:48859)
I20240503 14:14:00.110234 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 344eb942dc3d4188bc9baf3d1ecc4f7a from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:48859: session id = 6bbc686598f24f518d7ca6bb2063058d-344eb942dc3d4188bc9baf3d1ecc4f7a
I20240503 14:14:00.113737 28063 tablet_copy_source_session.cc:205] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.122753 30939 raft_consensus.cc:1203] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.126579 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.132474 924 tablet_copy_client.cc:783] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.132735 924 tablet_copy_client.cc:647] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.134555 924 tablet_copy_client.cc:523] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.141613 924 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:00.147511 924 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.148006 924 tablet_bootstrap.cc:492] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:00.148257 924 ts_tablet_manager.cc:1367] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:14:00.149019 924 raft_consensus.cc:348] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:00.149271 924 raft_consensus.cc:729] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.149535 924 consensus_queue.cc:260] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:00.150070 924 ts_tablet_manager.cc:1398] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:00.150799 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-344eb942dc3d4188bc9baf3d1ecc4f7a received from {username='slave'} at 127.20.240.70:48859
I20240503 14:14:00.150954 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-344eb942dc3d4188bc9baf3d1ecc4f7a on tablet 344eb942dc3d4188bc9baf3d1ecc4f7a with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:00.164083 976 ts_tablet_manager.cc:902] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:14:00.183048 28303 consensus_queue.cc:237] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:00.184242 976 tablet_copy_client.cc:308] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:14:00.185186 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet 9517b56a435b48e18c79d2a375e5a951 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:43505)
I20240503 14:14:00.185364 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet 9517b56a435b48e18c79d2a375e5a951 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:43505: session id = 793a0d5dd1634c5faaf0e1e4851e8294-9517b56a435b48e18c79d2a375e5a951
I20240503 14:14:00.188410 31092 tablet_copy_source_session.cc:205] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.191021 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9517b56a435b48e18c79d2a375e5a951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.193984 28433 raft_consensus.cc:1261] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.194907 784 consensus_queue.cc:1035] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:00.195544 28237 consensus_peers.cc:485] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dea30267166942258d47155fd0173713. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.196619 28171 raft_consensus.cc:1261] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.197394 949 consensus_queue.cc:1035] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:00.201203 976 tablet_copy_client.cc:783] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.201462 976 tablet_copy_client.cc:647] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.205562 976 tablet_copy_client.cc:523] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.206214 949 raft_consensus.cc:2932] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.207228 28433 raft_consensus.cc:2932] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.210121 976 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:14:00.212767 28173 raft_consensus.cc:2932] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.214290 27896 catalog_manager.cc:5484] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.227058 969 ts_tablet_manager.cc:902] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:00.235821 976 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.236403 976 tablet_bootstrap.cc:492] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:14:00.236737 976 ts_tablet_manager.cc:1367] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240503 14:14:00.237715 976 raft_consensus.cc:348] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:00.238108 976 raft_consensus.cc:729] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.238431 976 consensus_queue.cc:260] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:00.239028 976 ts_tablet_manager.cc:1398] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:00.239887 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-9517b56a435b48e18c79d2a375e5a951 received from {username='slave'} at 127.20.240.65:43505
I20240503 14:14:00.240031 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-9517b56a435b48e18c79d2a375e5a951 on tablet 9517b56a435b48e18c79d2a375e5a951 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:14:00.250615 969 tablet_copy_client.cc:308] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:00.251865 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 016a1c41112441619700c662beee8c98 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:54479)
I20240503 14:14:00.252035 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 016a1c41112441619700c662beee8c98 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:54479: session id = 6409961ee8c0445687c11724213c44a9-016a1c41112441619700c662beee8c98
I20240503 14:14:00.254997 30961 tablet_copy_source_session.cc:205] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.256260 969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016a1c41112441619700c662beee8c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.262054 969 tablet_copy_client.cc:783] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.262287 969 tablet_copy_client.cc:647] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.263958 969 tablet_copy_client.cc:523] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.266975 969 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:14:00.272928 969 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.273322 969 tablet_bootstrap.cc:492] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:14:00.273653 969 ts_tablet_manager.cc:1367] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240503 14:14:00.274577 969 raft_consensus.cc:348] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:00.274775 969 raft_consensus.cc:729] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.275002 969 consensus_queue.cc:260] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:00.275496 969 ts_tablet_manager.cc:1398] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:00.276461 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-016a1c41112441619700c662beee8c98 received from {username='slave'} at 127.20.240.66:54479
I20240503 14:14:00.276585 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 6409961ee8c0445687c11724213c44a9-016a1c41112441619700c662beee8c98 on tablet 016a1c41112441619700c662beee8c98 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:14:00.294325 28433 consensus_queue.cc:237] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:00.297017 28171 raft_consensus.cc:1261] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.297300 28302 raft_consensus.cc:1261] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.297631 826 consensus_queue.cc:1035] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:00.297943 771 consensus_queue.cc:1035] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:00.298556 28369 consensus_peers.cc:485] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49d89642d5364479877c7ba4700fdcee. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.300827 28043 raft_consensus.cc:1203] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.310914 826 raft_consensus.cc:2932] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:00.313534 28302 raft_consensus.cc:2932] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:00.316059 28171 raft_consensus.cc:2932] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:00.322049 27896 catalog_manager.cc:5484] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:00.348655 1092 ts_tablet_manager.cc:902] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:00.352097 28171 raft_consensus.cc:1203] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.392061 1092 tablet_copy_client.cc:308] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:00.393038 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet cd84de9286eb4ad7b88b9f243eb2801c from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:53837)
I20240503 14:14:00.393275 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet cd84de9286eb4ad7b88b9f243eb2801c from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:53837: session id = d0dbfb0e43f948c3922926aacc365fc1-cd84de9286eb4ad7b88b9f243eb2801c
I20240503 14:14:00.404712 28324 tablet_copy_source_session.cc:205] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.408030 924 ts_tablet_manager.cc:902] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:14:00.408360 1092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd84de9286eb4ad7b88b9f243eb2801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.415786 1092 tablet_copy_client.cc:783] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.416087 1092 tablet_copy_client.cc:647] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.423902 924 tablet_copy_client.cc:308] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:14:00.424558 1092 tablet_copy_client.cc:523] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.425011 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet f4b9d916f3fb4a348a8702662c143435 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:45791)
I20240503 14:14:00.425253 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet f4b9d916f3fb4a348a8702662c143435 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:45791: session id = 6bbc686598f24f518d7ca6bb2063058d-f4b9d916f3fb4a348a8702662c143435
I20240503 14:14:00.430297 1092 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:00.430707 30830 tablet_copy_source_session.cc:205] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.432488 28433 consensus_queue.cc:237] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:00.432668 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b9d916f3fb4a348a8702662c143435. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.442867 924 tablet_copy_client.cc:783] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.443367 924 tablet_copy_client.cc:647] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.445283 924 tablet_copy_client.cc:523] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.445576 28043 raft_consensus.cc:1261] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:00.445678 28369 consensus_peers.cc:485] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7ba7409f5bd4531bc5370019aca14c7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.445869 28302 raft_consensus.cc:1261] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.446566 826 consensus_queue.cc:1035] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:00.446856 1079 consensus_queue.cc:1035] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:00.449168 924 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:00.463091 826 raft_consensus.cc:2932] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:00.464043 924 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.464419 924 tablet_bootstrap.cc:492] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:00.464651 924 ts_tablet_manager.cc:1367] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20240503 14:14:00.465345 924 raft_consensus.cc:348] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:00.465559 924 raft_consensus.cc:729] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.465828 924 consensus_queue.cc:260] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:00.466004 28303 raft_consensus.cc:2932] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:00.466457 924 ts_tablet_manager.cc:1398] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:00.469242 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-f4b9d916f3fb4a348a8702662c143435 received from {username='slave'} at 127.20.240.70:45791
I20240503 14:14:00.470415 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-f4b9d916f3fb4a348a8702662c143435 on tablet f4b9d916f3fb4a348a8702662c143435 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:00.474392 27898 catalog_manager.cc:5484] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:00.474498 28043 raft_consensus.cc:2932] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:00.491861 969 ts_tablet_manager.cc:902] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:00.493948 976 ts_tablet_manager.cc:902] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:14:00.504304 28303 raft_consensus.cc:1203] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.552626 969 tablet_copy_client.cc:308] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:00.554965 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 9132d0f9d1ea4be49458a085c9e98a65 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:54479)
I20240503 14:14:00.555148 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 9132d0f9d1ea4be49458a085c9e98a65 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:54479: session id = 6409961ee8c0445687c11724213c44a9-9132d0f9d1ea4be49458a085c9e98a65
I20240503 14:14:00.557747 30938 raft_consensus.cc:1203] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.558050 30961 tablet_copy_source_session.cc:205] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.561494 969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9132d0f9d1ea4be49458a085c9e98a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.568630 976 tablet_copy_client.cc:308] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:14:00.569222 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet 618ba33946424065a465217bae7cdbb0 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:43505)
I20240503 14:14:00.569353 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet 618ba33946424065a465217bae7cdbb0 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:43505: session id = 793a0d5dd1634c5faaf0e1e4851e8294-618ba33946424065a465217bae7cdbb0
I20240503 14:14:00.571998 31092 tablet_copy_source_session.cc:205] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.574285 1092 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.575039 1092 tablet_bootstrap.cc:492] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:00.575400 1092 ts_tablet_manager.cc:1367] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.145s user 0.065s sys 0.018s
I20240503 14:14:00.576903 1092 raft_consensus.cc:348] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:00.577220 1092 raft_consensus.cc:729] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.577431 1092 consensus_queue.cc:260] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:00.578163 1092 ts_tablet_manager.cc:1398] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:14:00.578992 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-cd84de9286eb4ad7b88b9f243eb2801c received from {username='slave'} at 127.20.240.69:53837
I20240503 14:14:00.579200 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-cd84de9286eb4ad7b88b9f243eb2801c on tablet cd84de9286eb4ad7b88b9f243eb2801c with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:00.580932 969 tablet_copy_client.cc:783] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.581195 969 tablet_copy_client.cc:647] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.584450 969 tablet_copy_client.cc:523] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.584880 976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 618ba33946424065a465217bae7cdbb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.592016 969 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:14:00.592339 976 tablet_copy_client.cc:783] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.592569 976 tablet_copy_client.cc:647] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.595688 28171 raft_consensus.cc:1203] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.605093 976 tablet_copy_client.cc:523] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.606537 969 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.606926 969 tablet_bootstrap.cc:492] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:14:00.607157 969 ts_tablet_manager.cc:1367] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20240503 14:14:00.608501 976 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:14:00.608417 969 raft_consensus.cc:348] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:00.608630 969 raft_consensus.cc:729] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.608886 969 consensus_queue.cc:260] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:00.609452 969 ts_tablet_manager.cc:1398] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:00.610169 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-9132d0f9d1ea4be49458a085c9e98a65 received from {username='slave'} at 127.20.240.66:54479
I20240503 14:14:00.610294 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 6409961ee8c0445687c11724213c44a9-9132d0f9d1ea4be49458a085c9e98a65 on tablet 9132d0f9d1ea4be49458a085c9e98a65 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:14:00.631690 28040 raft_consensus.cc:1203] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.641551 976 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.641950 976 tablet_bootstrap.cc:492] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:14:00.642267 976 ts_tablet_manager.cc:1367] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.013s
I20240503 14:14:00.642958 976 raft_consensus.cc:348] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:00.643167 976 raft_consensus.cc:729] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.643369 976 consensus_queue.cc:260] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:00.643939 976 ts_tablet_manager.cc:1398] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:14:00.644932 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-618ba33946424065a465217bae7cdbb0 received from {username='slave'} at 127.20.240.65:43505
I20240503 14:14:00.645289 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-618ba33946424065a465217bae7cdbb0 on tablet 618ba33946424065a465217bae7cdbb0 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:14:00.672163 1092 ts_tablet_manager.cc:902] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:00.682022 30808 consensus_queue.cc:237] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:14:00.692785 30744 consensus_peers.cc:485] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9119ae1f5ca4c12ba7a4444edd210cf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.694049 31202 raft_consensus.cc:1261] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:00.695029 568 consensus_queue.cc:1035] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:00.702003 28043 raft_consensus.cc:1261] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:00.702785 568 consensus_queue.cc:1035] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:00.710017 1092 tablet_copy_client.cc:308] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:00.710052 868 raft_consensus.cc:2932] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.710671 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet dea30267166942258d47155fd0173713 from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:53837)
I20240503 14:14:00.710820 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet dea30267166942258d47155fd0173713 from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:53837: session id = d0dbfb0e43f948c3922926aacc365fc1-dea30267166942258d47155fd0173713
I20240503 14:14:00.710690 31202 raft_consensus.cc:2932] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.713613 28324 tablet_copy_source_session.cc:205] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.715674 28043 raft_consensus.cc:2932] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.718534 1092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea30267166942258d47155fd0173713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.720121 27898 catalog_manager.cc:5484] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:00.732519 1092 tablet_copy_client.cc:783] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.732786 1092 tablet_copy_client.cc:647] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.744397 1092 tablet_copy_client.cc:523] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.748603 1092 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:00.765789 30808 raft_consensus.cc:1203] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:00.794732 28171 consensus_queue.cc:237] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:00.795044 1092 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.795516 1092 tablet_bootstrap.cc:492] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:00.795778 1092 ts_tablet_manager.cc:1367] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.048s user 0.042s sys 0.002s
I20240503 14:14:00.796658 1092 raft_consensus.cc:348] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:00.796926 1092 raft_consensus.cc:729] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:00.797223 1092 consensus_queue.cc:260] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:00.797827 1092 ts_tablet_manager.cc:1398] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:00.798072 28303 raft_consensus.cc:1261] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.798487 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-dea30267166942258d47155fd0173713 received from {username='slave'} at 127.20.240.69:53837
I20240503 14:14:00.798532 28434 raft_consensus.cc:1261] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:00.798584 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-dea30267166942258d47155fd0173713 on tablet dea30267166942258d47155fd0173713 with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:00.799134 1168 consensus_queue.cc:1035] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:00.799438 761 consensus_queue.cc:1035] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:00.799854 28107 consensus_peers.cc:485] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30e4fb0a79814959aa25cf6eb34dfbb9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.807909 761 raft_consensus.cc:2932] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.808576 28303 raft_consensus.cc:2932] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.812578 28434 raft_consensus.cc:2932] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.815021 27893 catalog_manager.cc:5484] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:00.822209 30938 raft_consensus.cc:1203] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:00.854580 28434 consensus_queue.cc:237] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:00.857102 30938 raft_consensus.cc:1261] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:00.857141 28370 consensus_peers.cc:485] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 029be0389e244435bb08db2361729ad2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.857446 28043 raft_consensus.cc:1261] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:00.857614 826 consensus_queue.cc:1035] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:00.857880 771 consensus_queue.cc:1035] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:00.860412 771 raft_consensus.cc:2932] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:00.860971 28043 raft_consensus.cc:2932] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:00.865427 30939 raft_consensus.cc:2932] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:00.871184 27893 catalog_manager.cc:5484] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:00.905303 924 ts_tablet_manager.cc:902] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:00.938158 924 tablet_copy_client.cc:308] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:00.938827 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 49d89642d5364479877c7ba4700fdcee from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:50515)
I20240503 14:14:00.938963 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 49d89642d5364479877c7ba4700fdcee from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:50515: session id = 6bbc686598f24f518d7ca6bb2063058d-49d89642d5364479877c7ba4700fdcee
I20240503 14:14:00.941972 28454 tablet_copy_source_session.cc:205] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.943254 924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49d89642d5364479877c7ba4700fdcee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.948125 924 tablet_copy_client.cc:783] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.948457 924 tablet_copy_client.cc:647] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.952508 924 tablet_copy_client.cc:523] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.958603 924 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:00.969265 28173 consensus_queue.cc:237] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:00.971858 31203 raft_consensus.cc:1261] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:00.972432 31072 raft_consensus.cc:1261] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:00.972421 802 consensus_queue.cc:1035] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:00.973392 802 consensus_queue.cc:1035] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:00.974454 28110 consensus_peers.cc:485] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3905bce9ed6345f9979bda9500d1d053. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.980162 761 raft_consensus.cc:2932] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:00.980794 31203 raft_consensus.cc:2932] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:00.984611 31072 raft_consensus.cc:2932] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:00.986501 27898 catalog_manager.cc:5484] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:01.015607 28173 raft_consensus.cc:1203] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:01.020712 924 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:01.021117 924 tablet_bootstrap.cc:492] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:01.021368 924 ts_tablet_manager.cc:1367] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.063s user 0.043s sys 0.014s
I20240503 14:14:01.022089 924 raft_consensus.cc:348] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:01.022382 924 raft_consensus.cc:729] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:01.022624 924 consensus_queue.cc:260] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:01.023138 924 ts_tablet_manager.cc:1398] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:01.025115 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-49d89642d5364479877c7ba4700fdcee received from {username='slave'} at 127.20.240.70:50515
I20240503 14:14:01.025235 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-49d89642d5364479877c7ba4700fdcee on tablet 49d89642d5364479877c7ba4700fdcee with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:01.058178 28043 raft_consensus.cc:1203] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:01.063973 31203 consensus_queue.cc:237] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:01.070464 31070 raft_consensus.cc:1261] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:01.070497 28434 raft_consensus.cc:1261] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:01.071135 1229 ts_tablet_manager.cc:902] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:01.071339 643 consensus_queue.cc:1035] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:01.073889 31137 consensus_peers.cc:485] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fb8c05985f34deebc145c3c51dc413d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:01.075289 643 consensus_queue.cc:1035] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:01.109519 643 raft_consensus.cc:2932] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:01.114951 31070 raft_consensus.cc:2932] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:01.115996 28434 raft_consensus.cc:2932] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:01.119573 27896 catalog_manager.cc:5484] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:01.140539 1229 tablet_copy_client.cc:308] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:01.142601 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet d7ba7409f5bd4531bc5370019aca14c7 from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:60297)
I20240503 14:14:01.142783 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet d7ba7409f5bd4531bc5370019aca14c7 from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:60297: session id = fde134d0fa2b4900b9badd2e08fea09d-d7ba7409f5bd4531bc5370019aca14c7
I20240503 14:14:01.145098 28454 tablet_copy_source_session.cc:205] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.146819 1229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7ba7409f5bd4531bc5370019aca14c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.153270 1229 tablet_copy_client.cc:783] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.153657 1229 tablet_copy_client.cc:647] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.155831 1229 tablet_copy_client.cc:523] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.160184 1229 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:01.171089 30809 raft_consensus.cc:1203] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:01.205281 1250 ts_tablet_manager.cc:902] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:14:01.216527 1229 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:01.217162 1229 tablet_bootstrap.cc:492] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:01.217536 1229 ts_tablet_manager.cc:1367] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.057s user 0.049s sys 0.000s
I20240503 14:14:01.218650 1229 raft_consensus.cc:348] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:01.219025 1229 raft_consensus.cc:729] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:01.219368 1229 consensus_queue.cc:260] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:01.220098 1229 ts_tablet_manager.cc:1398] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
I20240503 14:14:01.220930 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-d7ba7409f5bd4531bc5370019aca14c7 received from {username='slave'} at 127.20.240.71:60297
I20240503 14:14:01.221045 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-d7ba7409f5bd4531bc5370019aca14c7 on tablet d7ba7409f5bd4531bc5370019aca14c7 with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:01.238415 1250 tablet_copy_client.cc:308] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:14:01.239107 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet d9119ae1f5ca4c12ba7a4444edd210cf from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:46535)
I20240503 14:14:01.239295 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet d9119ae1f5ca4c12ba7a4444edd210cf from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:46535: session id = 6409961ee8c0445687c11724213c44a9-d9119ae1f5ca4c12ba7a4444edd210cf
I20240503 14:14:01.243386 30830 tablet_copy_source_session.cc:205] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.245031 1250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9119ae1f5ca4c12ba7a4444edd210cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.250689 1250 tablet_copy_client.cc:783] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.250963 1250 tablet_copy_client.cc:647] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.251433 1092 ts_tablet_manager.cc:902] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:01.253507 1250 tablet_copy_client.cc:523] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.262799 1250 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:14:01.270999 1250 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:01.271540 1250 tablet_bootstrap.cc:492] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:14:01.271924 1250 ts_tablet_manager.cc:1367] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 14:14:01.272778 1250 raft_consensus.cc:348] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:01.273064 1250 raft_consensus.cc:729] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:14:01.273366 1250 consensus_queue.cc:260] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:01.278568 1092 tablet_copy_client.cc:308] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:01.279129 1250 ts_tablet_manager.cc:1398] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:14:01.279350 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet 30e4fb0a79814959aa25cf6eb34dfbb9 from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:47863)
I20240503 14:14:01.279466 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet 30e4fb0a79814959aa25cf6eb34dfbb9 from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:47863: session id = d0dbfb0e43f948c3922926aacc365fc1-30e4fb0a79814959aa25cf6eb34dfbb9
I20240503 14:14:01.279762 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-d9119ae1f5ca4c12ba7a4444edd210cf received from {username='slave'} at 127.20.240.66:46535
I20240503 14:14:01.279860 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session 6409961ee8c0445687c11724213c44a9-d9119ae1f5ca4c12ba7a4444edd210cf on tablet d9119ae1f5ca4c12ba7a4444edd210cf with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:14:01.282197 28194 tablet_copy_source_session.cc:205] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.283394 1092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e4fb0a79814959aa25cf6eb34dfbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.290378 1092 tablet_copy_client.cc:783] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.290695 1092 tablet_copy_client.cc:647] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.293257 1092 tablet_copy_client.cc:523] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.297988 1092 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:01.377264 1092 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:01.377269 30939 raft_consensus.cc:1203] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:01.377796 1092 tablet_bootstrap.cc:492] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:01.378114 1092 ts_tablet_manager.cc:1367] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.080s user 0.046s sys 0.008s
I20240503 14:14:01.378975 1092 raft_consensus.cc:348] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:01.379236 1092 raft_consensus.cc:729] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:01.379505 1092 consensus_queue.cc:260] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:01.380154 1092 ts_tablet_manager.cc:1398] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:01.380831 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-30e4fb0a79814959aa25cf6eb34dfbb9 received from {username='slave'} at 127.20.240.69:47863
I20240503 14:14:01.380954 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-30e4fb0a79814959aa25cf6eb34dfbb9 on tablet 30e4fb0a79814959aa25cf6eb34dfbb9 with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:01.395459 1286 ts_tablet_manager.cc:902] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:01.445222 31070 raft_consensus.cc:1203] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:01.467067 1286 tablet_copy_client.cc:308] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:01.467700 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 029be0389e244435bb08db2361729ad2 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:51853)
I20240503 14:14:01.467857 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 029be0389e244435bb08db2361729ad2 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:51853: session id = a4cdf05256a0407c8929ac43eb55d44f-029be0389e244435bb08db2361729ad2
I20240503 14:14:01.470710 28454 tablet_copy_source_session.cc:205] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.483731 1286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029be0389e244435bb08db2361729ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.498662 1286 tablet_copy_client.cc:783] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.498955 1286 tablet_copy_client.cc:647] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.506320 1286 tablet_copy_client.cc:523] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.510078 1286 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:01.541961 1286 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:01.542456 1286 tablet_bootstrap.cc:492] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:01.542724 1286 ts_tablet_manager.cc:1367] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.004s
I20240503 14:14:01.543409 1286 raft_consensus.cc:348] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:01.543642 1286 raft_consensus.cc:729] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:01.543916 1286 consensus_queue.cc:260] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:01.544557 1286 ts_tablet_manager.cc:1398] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:01.545405 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-029be0389e244435bb08db2361729ad2 received from {username='slave'} at 127.20.240.67:51853
I20240503 14:14:01.545524 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-029be0389e244435bb08db2361729ad2 on tablet 029be0389e244435bb08db2361729ad2 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:01.550864 28172 raft_consensus.cc:1203] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:01.583534 1302 ts_tablet_manager.cc:902] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:01.585289 1302 tablet_copy_client.cc:308] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:01.586092 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet 3905bce9ed6345f9979bda9500d1d053 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:50673)
I20240503 14:14:01.586369 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet 3905bce9ed6345f9979bda9500d1d053 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:50673: session id = 793a0d5dd1634c5faaf0e1e4851e8294-3905bce9ed6345f9979bda9500d1d053
I20240503 14:14:01.588670 28194 tablet_copy_source_session.cc:205] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.590173 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3905bce9ed6345f9979bda9500d1d053. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.596760 1302 tablet_copy_client.cc:783] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.597079 1302 tablet_copy_client.cc:647] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.599057 1302 tablet_copy_client.cc:523] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.605893 1302 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:14:01.653870 1302 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:01.654372 1302 tablet_bootstrap.cc:492] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:14:01.654633 1302 ts_tablet_manager.cc:1367] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.019s
I20240503 14:14:01.655438 1302 raft_consensus.cc:348] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:01.655652 1302 raft_consensus.cc:729] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:14:01.655879 1302 consensus_queue.cc:260] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:01.656395 1250 ts_tablet_manager.cc:902] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:14:01.662112 1302 ts_tablet_manager.cc:1398] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:14:01.663242 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-3905bce9ed6345f9979bda9500d1d053 received from {username='slave'} at 127.20.240.65:50673
I20240503 14:14:01.663373 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-3905bce9ed6345f9979bda9500d1d053 on tablet 3905bce9ed6345f9979bda9500d1d053 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:14:01.668159 30808 raft_consensus.cc:1203] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:01.682958 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:01.686617 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:01.700754 1250 tablet_copy_client.cc:308] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.72:41331
I20240503 14:14:01.701686 31223 tablet_copy_service.cc:140] P 530893a961af4b95b6b687fa5ed22f9d: Received BeginTabletCopySession request for tablet 3fb8c05985f34deebc145c3c51dc413d from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:44663)
I20240503 14:14:01.701951 31223 tablet_copy_service.cc:161] P 530893a961af4b95b6b687fa5ed22f9d: Beginning new tablet copy session on tablet 3fb8c05985f34deebc145c3c51dc413d from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:44663: session id = 6409961ee8c0445687c11724213c44a9-3fb8c05985f34deebc145c3c51dc413d
I20240503 14:14:01.705818 31223 tablet_copy_source_session.cc:205] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:01.714879 1250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb8c05985f34deebc145c3c51dc413d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:01.721256 1250 tablet_copy_client.cc:783] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:01.721544 1250 tablet_copy_client.cc:647] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:01.729280 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:01.740120 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:01.758355 1250 tablet_copy_client.cc:523] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:01.770375 1250 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
W20240503 14:14:01.782708 31007 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 48077.9us
I20240503 14:14:01.792379 1250 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:01.792779 1250 tablet_bootstrap.cc:492] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:14:01.793079 1250 ts_tablet_manager.cc:1367] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.001s
I20240503 14:14:01.804103 1250 raft_consensus.cc:348] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:01.804363 1250 raft_consensus.cc:729] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:14:01.804601 1250 consensus_queue.cc:260] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:01.805301 1250 ts_tablet_manager.cc:1398] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240503 14:14:01.810016 31223 tablet_copy_service.cc:342] P 530893a961af4b95b6b687fa5ed22f9d: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-3fb8c05985f34deebc145c3c51dc413d received from {username='slave'} at 127.20.240.66:44663
I20240503 14:14:01.810223 31223 tablet_copy_service.cc:433] P 530893a961af4b95b6b687fa5ed22f9d: ending tablet copy session 6409961ee8c0445687c11724213c44a9-3fb8c05985f34deebc145c3c51dc413d on tablet 3fb8c05985f34deebc145c3c51dc413d with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:14:01.819037 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:01.860567 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:01.894464 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:01.906040 28303 raft_consensus.cc:1203] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:01.930063 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:02.116735 28043 raft_consensus.cc:1203] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:02.269387 28173 raft_consensus.cc:1203] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:02.343554 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:02.418557 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:02.428396 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:02.433873 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:02.470639 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:02.493505 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:02.519033 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:02.522495 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:02.704957 31071 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951"
dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44024
I20240503 14:14:02.705273 31071 raft_consensus.cc:595] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:02.707916 1325 raft_consensus.cc:979] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: : Instructing follower 793a0d5dd1634c5faaf0e1e4851e8294 to start an election
I20240503 14:14:02.708099 1325 raft_consensus.cc:1067] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Signalling peer 793a0d5dd1634c5faaf0e1e4851e8294 to start an election
I20240503 14:14:02.709206 28043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951"
dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
from {username='slave'} at 127.20.240.71:59609
I20240503 14:14:02.709403 28043 raft_consensus.cc:482] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:02.709498 28043 raft_consensus.cc:3037] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:02.713112 28043 raft_consensus.cc:504] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:02.714336 31071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:14:02.714555 31071 raft_consensus.cc:3032] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:02.714488 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:14:02.714497 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9517b56a435b48e18c79d2a375e5a951" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:14:02.714643 31071 raft_consensus.cc:729] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:14:02.714677 28433 raft_consensus.cc:3037] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:02.714702 28172 raft_consensus.cc:3037] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:02.714954 28043 leader_election.cc:288] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:02.714951 31071 consensus_queue.cc:260] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:02.715440 31071 raft_consensus.cc:3037] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:02.717500 28172 raft_consensus.cc:2454] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:14:02.718705 28433 raft_consensus.cc:2454] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:14:02.718086 31071 raft_consensus.cc:2454] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:14:02.719256 27979 leader_election.cc:302] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:14:02.719521 1088 raft_consensus.cc:2781] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:02.719645 1088 raft_consensus.cc:686] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:14:02.719887 1088 consensus_queue.cc:237] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:02.729683 27899 catalog_manager.cc:5484] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:03.109483 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:03.112321 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:03.118232 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:03.156384 28433 raft_consensus.cc:1261] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:03.158100 1412 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:03.246460 28172 raft_consensus.cc:1261] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:03.246951 1412 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:03.249176 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:03.269665 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:03.287703 31071 raft_consensus.cc:1261] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:03.288758 1483 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:03.300806 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:03.368532 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:03.371966 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:03.949772 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:03.960495 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:03.988117 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:04.014582 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:04.094130 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:14:04.161216 31139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 103322us
I20240503 14:14:04.215898 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:14:04.222470 31006 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.72:41331, user_credentials={real_user=slave}} blocked reactor thread for 56213.3us
I20240503 14:14:04.256510 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:04.257063 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:04.436995 30938 consensus_queue.cc:237] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:04.439239 28171 raft_consensus.cc:1261] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:04.439452 31072 raft_consensus.cc:1261] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:04.439857 1573 consensus_queue.cc:1035] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:04.440510 1573 consensus_queue.cc:1035] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:04.442363 1149 raft_consensus.cc:2932] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:04.442983 28171 raft_consensus.cc:2932] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:04.443770 31071 raft_consensus.cc:2932] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:04.448103 27893 catalog_manager.cc:5484] T 9132d0f9d1ea4be49458a085c9e98a65 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:04.456636 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 9132d0f9d1ea4be49458a085c9e98a65 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:04.457532 1592 tablet_replica.cc:330] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:04.457825 1592 raft_consensus.cc:2229] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:04.458552 1592 raft_consensus.cc:2258] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:04.459373 1592 ts_tablet_manager.cc:1875] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:04.470252 1592 ts_tablet_manager.cc:1888] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:04.470391 1592 log.cc:1196] T 9132d0f9d1ea4be49458a085c9e98a65 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/9132d0f9d1ea4be49458a085c9e98a65
I20240503 14:14:04.471169 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 9132d0f9d1ea4be49458a085c9e98a65 (table rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]) successfully deleted
I20240503 14:14:04.840698 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:04.840916 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:04.884846 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:04.906401 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:04.953343 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:05.006296 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:05.009865 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:05.059522 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:05.200184 28434 consensus_queue.cc:237] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:05.202646 28043 raft_consensus.cc:1261] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:05.202733 28304 raft_consensus.cc:1261] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:05.203212 826 consensus_queue.cc:1035] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:05.204834 826 consensus_queue.cc:1035] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:05.210772 1490 raft_consensus.cc:2932] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:05.211356 28304 raft_consensus.cc:2932] T 029be0389e244435bb08db2361729ad2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:05.211971 28043 raft_consensus.cc:2932] T 029be0389e244435bb08db2361729ad2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:05.220314 27896 catalog_manager.cc:5484] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:05.244853 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 029be0389e244435bb08db2361729ad2 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:14:05.255160 1685 tablet_replica.cc:330] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:14:05.255378 1685 raft_consensus.cc:2229] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:05.255681 1685 raft_consensus.cc:2258] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:05.256438 1685 ts_tablet_manager.cc:1875] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:05.263722 1685 ts_tablet_manager.cc:1888] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:05.263844 1685 log.cc:1196] T 029be0389e244435bb08db2361729ad2 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/029be0389e244435bb08db2361729ad2
I20240503 14:14:05.264670 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 029be0389e244435bb08db2361729ad2 (table rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]) successfully deleted
W20240503 14:14:05.435267 28368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 46473.8us
I20240503 14:14:05.610000 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:05.643957 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:05.680393 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:05.737888 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:05.740458 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:05.769356 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:05.786761 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:05.855656 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:06.029575 28040 consensus_queue.cc:237] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:06.032225 28304 raft_consensus.cc:1261] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:06.032588 30938 raft_consensus.cc:1261] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:06.033099 1594 consensus_queue.cc:1035] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:06.034217 1594 consensus_queue.cc:1035] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:06.056377 1594 raft_consensus.cc:2932] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:06.059080 28304 raft_consensus.cc:2932] T 344eb942dc3d4188bc9baf3d1ecc4f7a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:06.059139 30938 raft_consensus.cc:2932] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:06.064054 27896 catalog_manager.cc:5484] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:06.097451 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 344eb942dc3d4188bc9baf3d1ecc4f7a with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:14:06.114742 1780 tablet_replica.cc:330] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:14:06.115191 1780 raft_consensus.cc:2229] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:06.115783 1780 raft_consensus.cc:2258] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:06.116613 1780 ts_tablet_manager.cc:1875] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:06.126184 1780 ts_tablet_manager.cc:1888] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:06.126435 1780 log.cc:1196] T 344eb942dc3d4188bc9baf3d1ecc4f7a P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/344eb942dc3d4188bc9baf3d1ecc4f7a
I20240503 14:14:06.128300 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 344eb942dc3d4188bc9baf3d1ecc4f7a (table rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]) successfully deleted
I20240503 14:14:06.503139 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:06.513010 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:06.546047 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:06.564112 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:06.583927 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:06.628710 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:06.638415 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:06.697690 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:06.837407 31071 consensus_queue.cc:237] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:06.839902 28302 raft_consensus.cc:1261] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:06.839902 30938 raft_consensus.cc:1261] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:06.840466 1326 consensus_queue.cc:1035] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:06.840827 1560 consensus_queue.cc:1035] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:06.843380 1560 raft_consensus.cc:2932] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:06.843811 28302 raft_consensus.cc:2932] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:06.850873 27897 catalog_manager.cc:5484] T a3888675106d4d389718380627c95edf P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:06.853921 30938 raft_consensus.cc:2932] T a3888675106d4d389718380627c95edf P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:06.860208 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet a3888675106d4d389718380627c95edf with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58086
I20240503 14:14:06.861088 1871 tablet_replica.cc:330] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:14:06.861462 1871 raft_consensus.cc:2229] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:06.861788 1871 raft_consensus.cc:2258] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:06.862740 1871 ts_tablet_manager.cc:1875] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:06.890337 1871 ts_tablet_manager.cc:1888] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:06.890599 1871 log.cc:1196] T a3888675106d4d389718380627c95edf P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/a3888675106d4d389718380627c95edf
I20240503 14:14:06.891439 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet a3888675106d4d389718380627c95edf (table rebalancer_extra_table_8 [id=7819d47059fa4ad0803d0feb733f35e3]) successfully deleted
I20240503 14:14:07.168756 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:07.179036 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:07.217252 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:07.245558 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:07.262569 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:07.281616 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:07.283327 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:07.286098 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:07.466866 28433 consensus_queue.cc:237] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:07.469918 31072 raft_consensus.cc:1261] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:07.472136 1717 consensus_queue.cc:1035] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:07.473229 28304 raft_consensus.cc:1261] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:07.473759 1717 consensus_queue.cc:1035] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:07.476389 826 raft_consensus.cc:2932] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:07.478621 28304 raft_consensus.cc:2932] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:07.488050 27894 catalog_manager.cc:5484] T d7ba7409f5bd4531bc5370019aca14c7 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:07.507901 31072 raft_consensus.cc:2932] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:07.518085 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet d7ba7409f5bd4531bc5370019aca14c7 with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52616
I20240503 14:14:07.534857 1954 tablet_replica.cc:330] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:14:07.535166 1954 raft_consensus.cc:2229] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:07.535514 1954 raft_consensus.cc:2258] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:07.536391 1954 ts_tablet_manager.cc:1875] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:07.547223 1954 ts_tablet_manager.cc:1888] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:07.547408 1954 log.cc:1196] T d7ba7409f5bd4531bc5370019aca14c7 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/d7ba7409f5bd4531bc5370019aca14c7
I20240503 14:14:07.548336 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet d7ba7409f5bd4531bc5370019aca14c7 (table rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]) successfully deleted
I20240503 14:14:08.095396 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:08.113862 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:08.125752 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:08.133472 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:08.169401 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:08.213672 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:08.256489 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:08.279274 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:08.419356 30938 consensus_queue.cc:237] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:08.421876 28302 raft_consensus.cc:1261] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:08.422765 28172 raft_consensus.cc:1261] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:08.423082 1573 consensus_queue.cc:1035] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:08.423969 1149 consensus_queue.cc:1035] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:08.428431 1573 raft_consensus.cc:2932] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:08.429200 28172 raft_consensus.cc:2932] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:08.431852 28302 raft_consensus.cc:2932] T a6b4ef3ee1fd440384272cee496fc102 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:08.435261 27896 catalog_manager.cc:5484] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:08.454331 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet a6b4ef3ee1fd440384272cee496fc102 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:08.455008 2048 tablet_replica.cc:330] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:08.455255 2048 raft_consensus.cc:2229] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:08.455507 2048 raft_consensus.cc:2258] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:08.456138 2048 ts_tablet_manager.cc:1875] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:08.466738 2048 ts_tablet_manager.cc:1888] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:08.466979 2048 log.cc:1196] T a6b4ef3ee1fd440384272cee496fc102 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/a6b4ef3ee1fd440384272cee496fc102
I20240503 14:14:08.468014 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet a6b4ef3ee1fd440384272cee496fc102 (table rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]) successfully deleted
I20240503 14:14:08.795616 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:08.796250 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:08.824718 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:08.832674 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:08.833410 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:08.834645 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:08.848928 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:08.903213 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:09.042110 28433 consensus_queue.cc:237] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:09.044221 28171 raft_consensus.cc:1261] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:09.044329 28040 raft_consensus.cc:1261] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:09.044754 2030 consensus_queue.cc:1035] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:09.045198 1947 consensus_queue.cc:1035] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:09.047693 1947 raft_consensus.cc:2932] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:09.048146 28171 raft_consensus.cc:2932] T 9a49047eedc54168af70a7b59fd7d770 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:09.051378 28040 raft_consensus.cc:2932] T 9a49047eedc54168af70a7b59fd7d770 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:09.055051 27900 catalog_manager.cc:5484] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:09.064870 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a49047eedc54168af70a7b59fd7d770 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:09.065761 2126 tablet_replica.cc:330] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:09.065963 2126 raft_consensus.cc:2229] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:09.066200 2126 raft_consensus.cc:2258] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:09.066792 2126 ts_tablet_manager.cc:1875] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:09.073359 2126 ts_tablet_manager.cc:1888] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:09.073542 2126 log.cc:1196] T 9a49047eedc54168af70a7b59fd7d770 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/9a49047eedc54168af70a7b59fd7d770
I20240503 14:14:09.074687 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 9a49047eedc54168af70a7b59fd7d770 (table rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]) successfully deleted
I20240503 14:14:09.431944 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:09.437825 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:09.442497 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:09.442677 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:09.472093 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:09.496289 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 14:14:09.552119 28238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 48341.9us
I20240503 14:14:09.570752 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:09.575302 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:09.752229 30939 consensus_queue.cc:237] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:09.754375 28172 raft_consensus.cc:1261] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:09.754417 28433 raft_consensus.cc:1261] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:09.754985 2198 consensus_queue.cc:1035] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:09.755270 1149 consensus_queue.cc:1035] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:09.758000 2037 raft_consensus.cc:2932] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:09.758642 28433 raft_consensus.cc:2932] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:09.758777 28172 raft_consensus.cc:2932] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:09.765719 27898 catalog_manager.cc:5484] T 016a1c41112441619700c662beee8c98 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:09.778030 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 016a1c41112441619700c662beee8c98 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:09.778710 2224 tablet_replica.cc:330] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:09.779318 2224 raft_consensus.cc:2229] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:09.779572 2224 raft_consensus.cc:2258] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:09.780244 2224 ts_tablet_manager.cc:1875] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:09.785724 2224 ts_tablet_manager.cc:1888] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:09.785828 2224 log.cc:1196] T 016a1c41112441619700c662beee8c98 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/016a1c41112441619700c662beee8c98
I20240503 14:14:09.786718 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 016a1c41112441619700c662beee8c98 (table rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]) successfully deleted
I20240503 14:14:10.180905 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:10.211664 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:10.236999 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:10.251856 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:10.252679 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:10.272234 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 14:14:10.290033 28108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.72:41331, user_credentials={real_user=slave}} blocked reactor thread for 67146.5us
W20240503 14:14:10.301010 28109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 78461us
I20240503 14:14:10.330693 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:10.342250 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:10.532043 28302 consensus_queue.cc:237] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:10.536021 28172 raft_consensus.cc:1261] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:10.536094 30808 raft_consensus.cc:1261] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:10.536579 1487 consensus_queue.cc:1035] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:10.536880 2199 consensus_queue.cc:1035] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:10.546923 2199 raft_consensus.cc:2932] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:10.550781 28172 raft_consensus.cc:2932] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:10.558923 27902 catalog_manager.cc:5484] T cd84de9286eb4ad7b88b9f243eb2801c P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:10.566854 30809 raft_consensus.cc:2932] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:10.580948 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet cd84de9286eb4ad7b88b9f243eb2801c with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58086
I20240503 14:14:10.592267 2318 tablet_replica.cc:330] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:14:10.592645 2318 raft_consensus.cc:2229] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:10.592958 2318 raft_consensus.cc:2258] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:10.593856 2318 ts_tablet_manager.cc:1875] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:10.598942 2318 ts_tablet_manager.cc:1888] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:10.599071 2318 log.cc:1196] T cd84de9286eb4ad7b88b9f243eb2801c P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/cd84de9286eb4ad7b88b9f243eb2801c
I20240503 14:14:10.599861 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet cd84de9286eb4ad7b88b9f243eb2801c (table rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]) successfully deleted
I20240503 14:14:10.870467 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:10.986912 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:11.000058 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:11.079900 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:11.104851 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:11.116290 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:11.147238 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:11.147451 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:11.291360 30808 consensus_queue.cc:237] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:11.293699 30941 raft_consensus.cc:1261] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:11.293766 31071 raft_consensus.cc:1261] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:11.294284 1727 consensus_queue.cc:1035] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:11.294786 1911 consensus_queue.cc:1035] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:11.297518 1911 raft_consensus.cc:2932] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:11.298166 30941 raft_consensus.cc:2932] T f4b9d916f3fb4a348a8702662c143435 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:11.299423 31071 raft_consensus.cc:2932] T f4b9d916f3fb4a348a8702662c143435 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:11.305837 27898 catalog_manager.cc:5484] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:11.313030 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b9d916f3fb4a348a8702662c143435 with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43626
I20240503 14:14:11.322624 2407 tablet_replica.cc:330] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:14:11.322849 2407 raft_consensus.cc:2229] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:11.323033 2407 raft_consensus.cc:2258] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:11.323544 2407 ts_tablet_manager.cc:1875] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:11.328845 2407 ts_tablet_manager.cc:1888] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:11.328953 2407 log.cc:1196] T f4b9d916f3fb4a348a8702662c143435 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4b9d916f3fb4a348a8702662c143435
I20240503 14:14:11.329605 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet f4b9d916f3fb4a348a8702662c143435 (table rebalancer_extra_table_64 [id=0d07d4d5f3eb44aa97ccd0c60127ab46]) successfully deleted
I20240503 14:14:11.714134 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:11.716693 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:11.731709 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:11.762060 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:11.763311 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:11.812218 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:11.829079 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:14:11.887264 31005 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 53601.9us
I20240503 14:14:11.947796 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:12.043641 31199 consensus_queue.cc:237] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:12.046794 28171 raft_consensus.cc:1261] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:12.046840 28434 raft_consensus.cc:1261] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:12.047634 2440 consensus_queue.cc:1035] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:12.048439 2440 consensus_queue.cc:1035] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:12.050877 1845 raft_consensus.cc:2932] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:12.051525 28434 raft_consensus.cc:2932] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:12.058339 27899 catalog_manager.cc:5484] T 3fb8c05985f34deebc145c3c51dc413d P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:12.072463 28171 raft_consensus.cc:2932] T 3fb8c05985f34deebc145c3c51dc413d P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:12.109035 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fb8c05985f34deebc145c3c51dc413d with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:12.109793 2505 tablet_replica.cc:330] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:12.110085 2505 raft_consensus.cc:2229] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:12.110476 2505 raft_consensus.cc:2258] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:12.113798 2505 ts_tablet_manager.cc:1875] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:12.124796 2505 ts_tablet_manager.cc:1888] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:12.124943 2505 log.cc:1196] T 3fb8c05985f34deebc145c3c51dc413d P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/3fb8c05985f34deebc145c3c51dc413d
I20240503 14:14:12.125694 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 3fb8c05985f34deebc145c3c51dc413d (table rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]) successfully deleted
I20240503 14:14:12.427806 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:12.448365 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:12.482211 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:12.571593 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:12.572002 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:12.579205 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:14:12.599229 28240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 45851.7us
I20240503 14:14:12.626509 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:12.634577 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:12.768908 31070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "618ba33946424065a465217bae7cdbb0"
dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40662
I20240503 14:14:12.769120 31070 raft_consensus.cc:595] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:12.816511 1326 raft_consensus.cc:979] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: : Instructing follower 6409961ee8c0445687c11724213c44a9 to start an election
I20240503 14:14:12.816701 1326 raft_consensus.cc:1067] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Signalling peer 6409961ee8c0445687c11724213c44a9 to start an election
I20240503 14:14:12.817368 28172 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "618ba33946424065a465217bae7cdbb0"
dest_uuid: "6409961ee8c0445687c11724213c44a9"
from {username='slave'} at 127.20.240.71:43001
I20240503 14:14:12.817636 28172 raft_consensus.cc:482] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:12.817726 28172 raft_consensus.cc:3037] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:12.820484 28172 raft_consensus.cc:504] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:12.822371 31199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ba33946424065a465217bae7cdbb0" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:14:12.822615 31199 raft_consensus.cc:3037] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:12.822891 28172 leader_election.cc:288] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:12.823312 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ba33946424065a465217bae7cdbb0" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:14:12.823490 28040 raft_consensus.cc:3037] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:12.825258 31199 raft_consensus.cc:2454] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:14:12.825838 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "618ba33946424065a465217bae7cdbb0" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:14:12.826058 31070 raft_consensus.cc:3032] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:12.826121 31070 raft_consensus.cc:729] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:14:12.826108 28040 raft_consensus.cc:2454] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:14:12.826392 31070 consensus_queue.cc:260] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:12.826745 31070 raft_consensus.cc:3037] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:12.827637 28110 leader_election.cc:302] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:14:12.827905 2378 raft_consensus.cc:2781] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:12.828011 2378 raft_consensus.cc:686] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:14:12.828313 2378 consensus_queue.cc:237] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:12.829474 31070 raft_consensus.cc:2454] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:14:12.833513 27895 catalog_manager.cc:5484] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 1 to 2, leader changed from fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:13.183338 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:13.190644 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:13.202140 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:13.226140 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:13.228704 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:13.239240 28040 raft_consensus.cc:1261] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:13.239856 2626 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:13.245294 31203 raft_consensus.cc:1261] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:13.246215 2626 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:13.289013 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:13.295547 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:13.318619 31070 raft_consensus.cc:1261] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:13.319540 802 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:13.341322 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:13.795318 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:13.805332 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:13.808270 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:13.831878 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:13.842660 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:13.851696 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:13.942061 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:14.007555 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:14.143966 28171 consensus_queue.cc:237] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:14.146550 30938 raft_consensus.cc:1261] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:14.146701 31199 raft_consensus.cc:1261] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:14.147260 2626 consensus_queue.cc:1035] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:14.147548 2378 consensus_queue.cc:1035] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:14.149861 2626 raft_consensus.cc:2932] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:14.153090 30938 raft_consensus.cc:2932] T 45da3dab7a204d378273a8985d91ace8 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:14.156607 27893 catalog_manager.cc:5484] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:14.172612 31199 raft_consensus.cc:2932] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:14.189296 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 45da3dab7a204d378273a8985d91ace8 with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:14:14.190153 2736 tablet_replica.cc:330] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:14:14.196019 2736 raft_consensus.cc:2229] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:14.196354 2736 raft_consensus.cc:2258] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:14.197144 2736 ts_tablet_manager.cc:1875] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:14.203763 2736 ts_tablet_manager.cc:1888] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:14.203884 2736 log.cc:1196] T 45da3dab7a204d378273a8985d91ace8 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/45da3dab7a204d378273a8985d91ace8
I20240503 14:14:14.204595 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 45da3dab7a204d378273a8985d91ace8 (table rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]) successfully deleted
I20240503 14:14:14.521376 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:14.551290 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:14.562569 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:14.577162 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:14.638171 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:14.662554 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:14.691716 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:14.748713 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:14.834190 28302 consensus_queue.cc:237] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:14.837175 30808 raft_consensus.cc:1261] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:14.837531 28173 raft_consensus.cc:1261] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:14.838276 2689 consensus_queue.cc:1035] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:14.838541 2559 consensus_queue.cc:1035] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:14.841434 2689 raft_consensus.cc:2932] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:14.842587 30808 raft_consensus.cc:2932] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:14.842890 28173 raft_consensus.cc:2932] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:14.848634 27902 catalog_manager.cc:5484] T dea30267166942258d47155fd0173713 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:14.853746 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet dea30267166942258d47155fd0173713 with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58086
I20240503 14:14:14.854403 2817 tablet_replica.cc:330] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:14:14.854612 2817 raft_consensus.cc:2229] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:14.854812 2817 raft_consensus.cc:2258] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:14.855561 2817 ts_tablet_manager.cc:1875] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:14.862946 2817 ts_tablet_manager.cc:1888] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:14.863070 2817 log.cc:1196] T dea30267166942258d47155fd0173713 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/dea30267166942258d47155fd0173713
I20240503 14:14:14.863935 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet dea30267166942258d47155fd0173713 (table rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]) successfully deleted
I20240503 14:14:15.224059 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:15.224658 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:15.239159 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:15.303074 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:15.364419 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:15.400756 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:15.477073 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:15.497908 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:15.730702 28171 consensus_queue.cc:237] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:15.732826 30808 raft_consensus.cc:1261] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:15.733193 28304 raft_consensus.cc:1261] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:15.733364 2749 consensus_queue.cc:1035] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:15.733704 2626 consensus_queue.cc:1035] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:15.736178 2626 raft_consensus.cc:2932] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:15.739008 30808 raft_consensus.cc:2932] T 30e4fb0a79814959aa25cf6eb34dfbb9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:15.741058 28304 raft_consensus.cc:2932] T 30e4fb0a79814959aa25cf6eb34dfbb9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:15.743641 27899 catalog_manager.cc:5484] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:15.761108 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e4fb0a79814959aa25cf6eb34dfbb9 with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58086
I20240503 14:14:15.762699 2904 tablet_replica.cc:330] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:14:15.762897 2904 raft_consensus.cc:2229] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:15.763125 2904 raft_consensus.cc:2258] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:15.763852 2904 ts_tablet_manager.cc:1875] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:15.775523 2904 ts_tablet_manager.cc:1888] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:15.775730 2904 log.cc:1196] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/30e4fb0a79814959aa25cf6eb34dfbb9
I20240503 14:14:15.776609 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 30e4fb0a79814959aa25cf6eb34dfbb9 (table rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]) successfully deleted
I20240503 14:14:16.072701 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:16.082464 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:16.108278 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:16.151114 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:16.198794 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:16.216713 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:16.247473 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:16.261445 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:16.422189 28432 consensus_queue.cc:237] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:16.424317 28172 raft_consensus.cc:1261] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:16.424352 30938 raft_consensus.cc:1261] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:16.424831 2385 consensus_queue.cc:1035] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:16.425349 2858 consensus_queue.cc:1035] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:16.428447 2858 raft_consensus.cc:2932] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:16.429329 28172 raft_consensus.cc:2932] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:16.437834 27902 catalog_manager.cc:5484] T 49d89642d5364479877c7ba4700fdcee P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:16.446729 30938 raft_consensus.cc:2932] T 49d89642d5364479877c7ba4700fdcee P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:16.453347 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 49d89642d5364479877c7ba4700fdcee with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43626
I20240503 14:14:16.468963 2985 tablet_replica.cc:330] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:14:16.469300 2985 raft_consensus.cc:2229] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:16.469609 2985 raft_consensus.cc:2258] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:16.470422 2985 ts_tablet_manager.cc:1875] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:16.475152 2985 ts_tablet_manager.cc:1888] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:16.475322 2985 log.cc:1196] T 49d89642d5364479877c7ba4700fdcee P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/49d89642d5364479877c7ba4700fdcee
I20240503 14:14:16.476579 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 49d89642d5364479877c7ba4700fdcee (table rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]) successfully deleted
I20240503 14:14:16.908226 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:16.929374 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:16.932466 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:16.959043 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:16.989867 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:17.007014 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:17.012454 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:17.067859 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:17.183617 30809 consensus_queue.cc:237] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:17.186338 28171 raft_consensus.cc:1261] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:17.186640 28041 raft_consensus.cc:1261] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:17.186916 2991 consensus_queue.cc:1035] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:17.187158 2400 consensus_queue.cc:1035] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:17.189949 2991 raft_consensus.cc:2932] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:17.190390 28171 raft_consensus.cc:2932] T d9119ae1f5ca4c12ba7a4444edd210cf P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:17.190639 28041 raft_consensus.cc:2932] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:17.197450 27898 catalog_manager.cc:5484] T d9119ae1f5ca4c12ba7a4444edd210cf P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:17.202353 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet d9119ae1f5ca4c12ba7a4444edd210cf with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:17.202981 3079 tablet_replica.cc:330] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:17.203187 3079 raft_consensus.cc:2229] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:17.203374 3079 raft_consensus.cc:2258] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:17.203892 3079 ts_tablet_manager.cc:1875] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:17.208731 3079 ts_tablet_manager.cc:1888] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:17.208842 3079 log.cc:1196] T d9119ae1f5ca4c12ba7a4444edd210cf P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/d9119ae1f5ca4c12ba7a4444edd210cf
I20240503 14:14:17.209410 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet d9119ae1f5ca4c12ba7a4444edd210cf (table rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]) successfully deleted
I20240503 14:14:17.744282 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:17.751259 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:17.772890 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:17.814546 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:17.862884 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:17.869274 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:17.899117 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:17.958189 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:18.045509 28171 consensus_queue.cc:237] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:18.047330 28041 raft_consensus.cc:1261] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:18.047627 31069 raft_consensus.cc:1261] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:18.048075 3033 consensus_queue.cc:1035] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:18.048323 2626 consensus_queue.cc:1035] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:18.051297 2626 raft_consensus.cc:2932] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.052064 28041 raft_consensus.cc:2932] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.054826 31069 raft_consensus.cc:2932] T 3905bce9ed6345f9979bda9500d1d053 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.059216 27894 catalog_manager.cc:5484] T 3905bce9ed6345f9979bda9500d1d053 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.075963 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 3905bce9ed6345f9979bda9500d1d053 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:18.077114 3178 tablet_replica.cc:330] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:18.077347 3178 raft_consensus.cc:2229] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:18.077572 3178 raft_consensus.cc:2258] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:18.078279 3178 ts_tablet_manager.cc:1875] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:18.085496 3178 ts_tablet_manager.cc:1888] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:18.085593 3178 log.cc:1196] T 3905bce9ed6345f9979bda9500d1d053 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/3905bce9ed6345f9979bda9500d1d053
I20240503 14:14:18.086303 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 3905bce9ed6345f9979bda9500d1d053 (table rebalancer_extra_table_22 [id=2498fceee67f406a8a21f9f8c19b426e]) successfully deleted
I20240503 14:14:18.493270 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:18.497911 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:18.615621 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:18.634944 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:18.636885 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:18.637862 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:18.638520 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:18.663193 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:18.843755 28041 consensus_queue.cc:237] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:18.845826 28171 raft_consensus.cc:1261] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:18.846024 28434 raft_consensus.cc:1261] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:18.846427 2379 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:18.846702 2686 consensus_queue.cc:1035] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:18.850179 2686 raft_consensus.cc:2932] T 9517b56a435b48e18c79d2a375e5a951 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.850734 28171 raft_consensus.cc:2932] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.854077 28434 raft_consensus.cc:2932] T 9517b56a435b48e18c79d2a375e5a951 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.860715 27894 catalog_manager.cc:5484] T 9517b56a435b48e18c79d2a375e5a951 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:18.875692 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 9517b56a435b48e18c79d2a375e5a951 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:18.876250 3276 tablet_replica.cc:330] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:18.876461 3276 raft_consensus.cc:2229] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:18.876745 3276 raft_consensus.cc:2258] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:18.877440 3276 ts_tablet_manager.cc:1875] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:18.883375 3276 ts_tablet_manager.cc:1888] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:14:18.883540 3276 log.cc:1196] T 9517b56a435b48e18c79d2a375e5a951 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/9517b56a435b48e18c79d2a375e5a951
I20240503 14:14:18.885068 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 9517b56a435b48e18c79d2a375e5a951 (table rebalancer_dynamic_table [id=168110ceedbf4e228913b8743651cb17]) successfully deleted
I20240503 14:14:19.329142 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:19.347674 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:19.377835 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:19.420871 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:19.421252 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:19.440240 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:19.448513 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:19.477860 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:19.684048 28171 consensus_queue.cc:237] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:19.686300 28042 raft_consensus.cc:1261] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:19.686378 31199 raft_consensus.cc:1261] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:19.686877 2626 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:19.687144 3033 consensus_queue.cc:1035] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:19.690780 2626 raft_consensus.cc:2932] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:19.691874 31199 raft_consensus.cc:2932] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:19.694801 28042 raft_consensus.cc:2932] T 618ba33946424065a465217bae7cdbb0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:19.698652 27895 catalog_manager.cc:5484] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:19.707574 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 618ba33946424065a465217bae7cdbb0 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:19.708161 3413 tablet_replica.cc:330] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:19.708408 3413 raft_consensus.cc:2229] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:19.708585 3413 raft_consensus.cc:2258] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:19.709203 3413 ts_tablet_manager.cc:1875] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:19.715402 3413 ts_tablet_manager.cc:1888] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:14:19.715585 3413 log.cc:1196] T 618ba33946424065a465217bae7cdbb0 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/618ba33946424065a465217bae7cdbb0
I20240503 14:14:19.716346 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 618ba33946424065a465217bae7cdbb0 (table rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]) successfully deleted
I20240503 14:14:19.819972 30939 consensus_queue.cc:237] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:19.822499 28434 raft_consensus.cc:1261] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:19.822599 30874 consensus_peers.cc:485] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9088c0106bf249cab5b35c1171a7e7a0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:19.822613 31199 raft_consensus.cc:1261] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:19.823163 3451 consensus_queue.cc:1035] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:19.823479 3248 consensus_queue.cc:1035] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:19.828110 2938 raft_consensus.cc:2932] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:19.829020 31199 raft_consensus.cc:2932] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:19.833410 28434 raft_consensus.cc:2932] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:19.838831 27902 catalog_manager.cc:5484] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:19.872838 31069 consensus_queue.cc:237] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:19.876639 30808 raft_consensus.cc:1261] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:19.877390 30939 raft_consensus.cc:1261] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:19.878329 1326 consensus_queue.cc:1035] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:19.878597 3142 consensus_queue.cc:1035] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:19.880591 31008 consensus_peers.cc:485] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e57c283674c342169940a5903516ec93. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:19.896147 1326 raft_consensus.cc:2932] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:19.896658 30808 raft_consensus.cc:2932] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:19.903952 27897 catalog_manager.cc:5484] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:19.908903 30939 raft_consensus.cc:2932] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:19.968384 28302 consensus_queue.cc:237] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:19.970710 28434 raft_consensus.cc:1261] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:19.971006 28238 consensus_peers.cc:485] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7b192797b6c40d4853e6ae4583ef3ca. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:19.971448 2559 consensus_queue.cc:1035] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:19.971540 28171 raft_consensus.cc:1261] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:19.972062 2559 consensus_queue.cc:1035] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:19.974771 3492 raft_consensus.cc:2932] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:19.975317 28171 raft_consensus.cc:2932] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:19.981459 28434 raft_consensus.cc:2932] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:19.984803 27897 catalog_manager.cc:5484] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:20.025370 28302 consensus_queue.cc:237] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:20.027340 28434 raft_consensus.cc:1261] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:20.027509 28173 raft_consensus.cc:1261] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:20.027647 28237 consensus_peers.cc:485] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bb01c0be7da4fe4b9c7c7160df7debc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.028034 3492 consensus_queue.cc:1035] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:20.028327 2559 consensus_queue.cc:1035] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:20.031080 2559 raft_consensus.cc:2932] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:20.031747 28173 raft_consensus.cc:2932] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:20.034611 28434 raft_consensus.cc:2932] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:20.036689 27897 catalog_manager.cc:5484] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:20.107247 31069 consensus_queue.cc:237] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:20.109849 30939 raft_consensus.cc:1261] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.109879 28173 raft_consensus.cc:1261] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:20.109897 31008 consensus_peers.cc:485] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e571695c819e485483ece09278ee1e09. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.110455 1326 consensus_queue.cc:1035] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.110891 1326 consensus_queue.cc:1035] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.128770 1326 raft_consensus.cc:2932] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.129585 30939 raft_consensus.cc:2932] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.136034 28173 raft_consensus.cc:2932] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.140164 27902 catalog_manager.cc:5484] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.186623 28434 consensus_queue.cc:237] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:20.188964 28173 raft_consensus.cc:1261] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:20.188957 28302 raft_consensus.cc:1261] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:20.189499 2858 consensus_queue.cc:1035] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:20.189854 28369 consensus_peers.cc:485] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48266bf4ad9b4976b639553490ffebbe. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.190109 2385 consensus_queue.cc:1035] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:20.192898 2385 raft_consensus.cc:2932] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:20.193490 28173 raft_consensus.cc:2932] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:20.196756 28302 raft_consensus.cc:2932] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:20.198765 27895 catalog_manager.cc:5484] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:20.272969 28173 consensus_queue.cc:237] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
W20240503 14:14:20.275570 28110 consensus_peers.cc:485] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da7bb0a1956a452cbc1cbbb3999fd7e2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.275592 31199 raft_consensus.cc:1261] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.275732 28042 raft_consensus.cc:1261] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.276103 2626 consensus_queue.cc:1035] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.276331 3526 consensus_queue.cc:1035] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.283409 2626 raft_consensus.cc:2932] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.283979 28042 raft_consensus.cc:2932] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.287400 31199 raft_consensus.cc:2932] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.289999 27895 catalog_manager.cc:5484] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.305166 3578 ts_tablet_manager.cc:902] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:20.306478 3578 tablet_copy_client.cc:308] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:20.307107 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 9088c0106bf249cab5b35c1171a7e7a0 from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:49233)
I20240503 14:14:20.307251 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 9088c0106bf249cab5b35c1171a7e7a0 from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:49233: session id = d0dbfb0e43f948c3922926aacc365fc1-9088c0106bf249cab5b35c1171a7e7a0
I20240503 14:14:20.309700 30961 tablet_copy_source_session.cc:205] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.310802 3578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9088c0106bf249cab5b35c1171a7e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.317190 3578 tablet_copy_client.cc:783] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.317371 3578 tablet_copy_client.cc:647] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.318892 3578 tablet_copy_client.cc:523] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.320933 3578 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:20.326490 3578 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.326941 3578 tablet_bootstrap.cc:492] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:20.327185 3578 ts_tablet_manager.cc:1367] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.005s
I20240503 14:14:20.328434 3578 raft_consensus.cc:348] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:20.328727 3578 raft_consensus.cc:729] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.328977 3578 consensus_queue.cc:260] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:20.329704 3578 ts_tablet_manager.cc:1398] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:20.330502 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-9088c0106bf249cab5b35c1171a7e7a0 received from {username='slave'} at 127.20.240.69:49233
I20240503 14:14:20.330606 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-9088c0106bf249cab5b35c1171a7e7a0 on tablet 9088c0106bf249cab5b35c1171a7e7a0 with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:20.334594 3594 ts_tablet_manager.cc:902] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:14:20.342381 3594 tablet_copy_client.cc:308] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:14:20.343509 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet e57c283674c342169940a5903516ec93 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:54103)
I20240503 14:14:20.343632 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet e57c283674c342169940a5903516ec93 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:54103: session id = a4cdf05256a0407c8929ac43eb55d44f-e57c283674c342169940a5903516ec93
I20240503 14:14:20.346307 31092 tablet_copy_source_session.cc:205] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.347244 31069 consensus_queue.cc:237] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:20.350368 3594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57c283674c342169940a5903516ec93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.350745 30808 raft_consensus.cc:1261] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:20.351151 31007 consensus_peers.cc:485] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e1fbc1d1bb6439c9f8b64364ad9ab4b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.351702 1326 consensus_queue.cc:1035] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.353013 28173 raft_consensus.cc:1261] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.353457 1326 consensus_queue.cc:1035] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.363420 3594 tablet_copy_client.cc:783] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.363970 3594 tablet_copy_client.cc:647] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.366977 3594 tablet_copy_client.cc:523] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.372382 3594 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:20.378096 3474 raft_consensus.cc:2932] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:20.383023 30808 raft_consensus.cc:2932] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:20.382524 28173 raft_consensus.cc:2932] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:20.385159 27895 catalog_manager.cc:5484] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:20.385807 3594 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.386315 3594 tablet_bootstrap.cc:492] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:20.386541 3594 ts_tablet_manager.cc:1367] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.003s
I20240503 14:14:20.387164 3594 raft_consensus.cc:348] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:20.387377 3594 raft_consensus.cc:729] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.387701 3594 consensus_queue.cc:260] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:20.390039 3594 ts_tablet_manager.cc:1398] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:20.390784 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-e57c283674c342169940a5903516ec93 received from {username='slave'} at 127.20.240.67:54103
I20240503 14:14:20.390894 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-e57c283674c342169940a5903516ec93 on tablet e57c283674c342169940a5903516ec93 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:20.433168 3611 ts_tablet_manager.cc:902] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:20.446067 3611 tablet_copy_client.cc:308] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:20.447027 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet d7b192797b6c40d4853e6ae4583ef3ca from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:38939)
I20240503 14:14:20.447160 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet d7b192797b6c40d4853e6ae4583ef3ca from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:38939: session id = 530893a961af4b95b6b687fa5ed22f9d-d7b192797b6c40d4853e6ae4583ef3ca
I20240503 14:14:20.450040 28324 tablet_copy_source_session.cc:205] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.451751 3611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b192797b6c40d4853e6ae4583ef3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.457871 3611 tablet_copy_client.cc:783] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.458130 3611 tablet_copy_client.cc:647] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.460151 3611 tablet_copy_client.cc:523] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.463887 3611 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:14:20.482797 30939 consensus_queue.cc:237] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
W20240503 14:14:20.485706 30877 consensus_peers.cc:485] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bb1ec320c4114eec8915b7f2bcffc2c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.485857 28173 raft_consensus.cc:1261] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.486677 3248 consensus_queue.cc:1035] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.486774 31069 raft_consensus.cc:1261] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.488027 3248 consensus_queue.cc:1035] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.522435 3611 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.522861 3611 tablet_bootstrap.cc:492] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:14:20.523097 3611 ts_tablet_manager.cc:1367] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.059s user 0.050s sys 0.006s
I20240503 14:14:20.523679 3611 raft_consensus.cc:348] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:20.523856 3611 raft_consensus.cc:729] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.524083 3611 consensus_queue.cc:260] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:20.524608 3611 ts_tablet_manager.cc:1398] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:14:20.525367 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-d7b192797b6c40d4853e6ae4583ef3ca received from {username='slave'} at 127.20.240.72:38939
I20240503 14:14:20.525466 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-d7b192797b6c40d4853e6ae4583ef3ca on tablet d7b192797b6c40d4853e6ae4583ef3ca with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:14:20.526400 3248 raft_consensus.cc:2932] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.536970 28172 raft_consensus.cc:2932] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.544878 31069 raft_consensus.cc:2932] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.550473 27901 catalog_manager.cc:5484] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:20.579877 3578 ts_tablet_manager.cc:902] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:20.582298 3578 tablet_copy_client.cc:308] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:20.582880 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 1bb01c0be7da4fe4b9c7c7160df7debc from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:53837)
I20240503 14:14:20.583029 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 1bb01c0be7da4fe4b9c7c7160df7debc from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:53837: session id = d0dbfb0e43f948c3922926aacc365fc1-1bb01c0be7da4fe4b9c7c7160df7debc
I20240503 14:14:20.585875 28324 tablet_copy_source_session.cc:205] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.587240 3578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bb01c0be7da4fe4b9c7c7160df7debc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.588794 3594 ts_tablet_manager.cc:902] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:14:20.593611 3578 tablet_copy_client.cc:783] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.593830 3578 tablet_copy_client.cc:647] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.594905 3594 tablet_copy_client.cc:308] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:14:20.595724 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet e571695c819e485483ece09278ee1e09 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:54103)
I20240503 14:14:20.595911 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet e571695c819e485483ece09278ee1e09 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:54103: session id = a4cdf05256a0407c8929ac43eb55d44f-e571695c819e485483ece09278ee1e09
I20240503 14:14:20.595990 3578 tablet_copy_client.cc:523] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.598134 3578 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:20.599720 31092 tablet_copy_source_session.cc:205] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.600890 3594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e571695c819e485483ece09278ee1e09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.606564 3594 tablet_copy_client.cc:783] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.606799 3594 tablet_copy_client.cc:647] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.608117 3594 tablet_copy_client.cc:523] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.611061 3594 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:20.615789 3594 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.616103 3594 tablet_bootstrap.cc:492] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:20.616324 3594 ts_tablet_manager.cc:1367] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:14:20.616847 3594 raft_consensus.cc:348] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:20.617031 3594 raft_consensus.cc:729] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.617249 3594 consensus_queue.cc:260] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:20.617792 3594 ts_tablet_manager.cc:1398] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:20.618510 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-e571695c819e485483ece09278ee1e09 received from {username='slave'} at 127.20.240.67:54103
I20240503 14:14:20.618613 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-e571695c819e485483ece09278ee1e09 on tablet e571695c819e485483ece09278ee1e09 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:20.663106 3657 ts_tablet_manager.cc:902] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:20.671938 28434 consensus_queue.cc:237] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:20.685293 31199 raft_consensus.cc:1261] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.685577 30939 raft_consensus.cc:1261] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:20.686326 28370 consensus_peers.cc:485] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b96b54a566db4bd1a31e8fb30096c942. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.686476 2858 consensus_queue.cc:1035] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.686995 2858 consensus_queue.cc:1035] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.705188 3578 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.705724 3578 tablet_bootstrap.cc:492] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:20.706018 3578 ts_tablet_manager.cc:1367] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.108s user 0.051s sys 0.001s
I20240503 14:14:20.707398 2858 raft_consensus.cc:2932] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:20.708150 31199 raft_consensus.cc:2932] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:20.712582 30939 raft_consensus.cc:2932] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:20.713495 3578 raft_consensus.cc:348] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:20.713716 3578 raft_consensus.cc:729] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.713904 3578 consensus_queue.cc:260] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:20.715090 27901 catalog_manager.cc:5484] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:20.715368 3578 ts_tablet_manager.cc:1398] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240503 14:14:20.716414 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-1bb01c0be7da4fe4b9c7c7160df7debc received from {username='slave'} at 127.20.240.69:53837
I20240503 14:14:20.716524 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-1bb01c0be7da4fe4b9c7c7160df7debc on tablet 1bb01c0be7da4fe4b9c7c7160df7debc with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:20.718134 3657 tablet_copy_client.cc:308] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:20.719985 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet da7bb0a1956a452cbc1cbbb3999fd7e2 from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:42383)
I20240503 14:14:20.720124 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet da7bb0a1956a452cbc1cbbb3999fd7e2 from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:42383: session id = 5388902973134958a030fd63ab4faba4-da7bb0a1956a452cbc1cbbb3999fd7e2
I20240503 14:14:20.722317 28194 tablet_copy_source_session.cc:205] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.724084 3657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da7bb0a1956a452cbc1cbbb3999fd7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.727165 3665 ts_tablet_manager.cc:902] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:20.734650 3657 tablet_copy_client.cc:783] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.734974 3657 tablet_copy_client.cc:647] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.736996 3657 tablet_copy_client.cc:523] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.742367 3657 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:20.759625 3657 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.760087 3657 tablet_bootstrap.cc:492] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:20.760901 3657 ts_tablet_manager.cc:1367] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20240503 14:14:20.761979 3657 raft_consensus.cc:348] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:20.762193 3657 raft_consensus.cc:729] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.762528 3657 consensus_queue.cc:260] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:20.763219 3657 ts_tablet_manager.cc:1398] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:20.764010 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-da7bb0a1956a452cbc1cbbb3999fd7e2 received from {username='slave'} at 127.20.240.68:42383
I20240503 14:14:20.764117 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 5388902973134958a030fd63ab4faba4-da7bb0a1956a452cbc1cbbb3999fd7e2 on tablet da7bb0a1956a452cbc1cbbb3999fd7e2 with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:20.771570 3665 tablet_copy_client.cc:308] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:20.775347 30808 raft_consensus.cc:1203] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:20.778990 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 48266bf4ad9b4976b639553490ffebbe from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:60297)
I20240503 14:14:20.779134 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 48266bf4ad9b4976b639553490ffebbe from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:60297: session id = fde134d0fa2b4900b9badd2e08fea09d-48266bf4ad9b4976b639553490ffebbe
I20240503 14:14:20.783793 28454 tablet_copy_source_session.cc:205] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.784947 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48266bf4ad9b4976b639553490ffebbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.790660 3665 tablet_copy_client.cc:783] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.790877 3665 tablet_copy_client.cc:647] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.802515 3665 tablet_copy_client.cc:523] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.806165 3665 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:20.839951 3683 ts_tablet_manager.cc:902] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:14:20.854322 28303 consensus_queue.cc:237] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:20.858000 31069 raft_consensus.cc:1261] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:20.859759 30939 raft_consensus.cc:1261] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:20.859895 28240 consensus_peers.cc:485] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8377b1aa704c43b78f310e7440f6488e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.863101 3080 consensus_queue.cc:1035] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.863605 3080 consensus_queue.cc:1035] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:20.865053 3683 tablet_copy_client.cc:308] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:14:20.865810 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:54391)
I20240503 14:14:20.865936 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:54391: session id = 6bbc686598f24f518d7ca6bb2063058d-2e1fbc1d1bb6439c9f8b64364ad9ab4b
I20240503 14:14:20.868750 31092 tablet_copy_source_session.cc:205] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:20.873131 3683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:20.876384 3665 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.876777 3665 tablet_bootstrap.cc:492] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:20.877013 3665 ts_tablet_manager.cc:1367] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.071s user 0.059s sys 0.000s
I20240503 14:14:20.877151 3080 raft_consensus.cc:2932] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.877743 3665 raft_consensus.cc:348] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:20.877934 3665 raft_consensus.cc:729] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.878132 3665 consensus_queue.cc:260] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:20.878928 3665 ts_tablet_manager.cc:1398] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:20.879570 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-48266bf4ad9b4976b639553490ffebbe received from {username='slave'} at 127.20.240.71:60297
I20240503 14:14:20.879688 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-48266bf4ad9b4976b639553490ffebbe on tablet 48266bf4ad9b4976b639553490ffebbe with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:20.883011 30941 raft_consensus.cc:2932] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.883881 31069 raft_consensus.cc:2932] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.887639 27897 catalog_manager.cc:5484] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:20.903527 28303 raft_consensus.cc:1203] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:20.906431 3683 tablet_copy_client.cc:783] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:20.911787 3683 tablet_copy_client.cc:647] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:20.915352 3683 tablet_copy_client.cc:523] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:20.924983 3683 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:20.940891 3683 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:20.941313 3683 tablet_bootstrap.cc:492] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:20.941533 3683 ts_tablet_manager.cc:1367] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240503 14:14:20.942109 3683 raft_consensus.cc:348] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:20.942312 3683 raft_consensus.cc:729] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:20.942612 3683 consensus_queue.cc:260] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:20.943193 3683 ts_tablet_manager.cc:1398] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:20.944065 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-2e1fbc1d1bb6439c9f8b64364ad9ab4b received from {username='slave'} at 127.20.240.70:54391
I20240503 14:14:20.944185 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-2e1fbc1d1bb6439c9f8b64364ad9ab4b on tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:20.972951 31199 raft_consensus.cc:1203] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:20.980590 3594 ts_tablet_manager.cc:902] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:20.993086 28434 consensus_queue.cc:237] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:20.995960 28172 raft_consensus.cc:1261] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:20.996363 28303 raft_consensus.cc:1261] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:20.996479 28367 consensus_peers.cc:485] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69d4d86cb68646bc9141c1e6df964561. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:20.997519 2385 consensus_queue.cc:1035] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:20.997864 2858 consensus_queue.cc:1035] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:20.999935 3594 tablet_copy_client.cc:308] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:21.000676 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet bb1ec320c4114eec8915b7f2bcffc2c3 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:54091)
I20240503 14:14:21.000820 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet bb1ec320c4114eec8915b7f2bcffc2c3 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:54091: session id = a4cdf05256a0407c8929ac43eb55d44f-bb1ec320c4114eec8915b7f2bcffc2c3
I20240503 14:14:21.000783 2385 raft_consensus.cc:2932] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.001361 28303 raft_consensus.cc:2932] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.002768 30961 tablet_copy_source_session.cc:205] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.003854 3594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1ec320c4114eec8915b7f2bcffc2c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.007879 27901 catalog_manager.cc:5484] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.011246 28171 raft_consensus.cc:2932] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.012429 3594 tablet_copy_client.cc:783] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.012681 3594 tablet_copy_client.cc:647] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.014689 3594 tablet_copy_client.cc:523] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.019549 3594 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:21.032586 3594 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.032943 3594 tablet_bootstrap.cc:492] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:21.033169 3594 ts_tablet_manager.cc:1367] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240503 14:14:21.033850 3594 raft_consensus.cc:348] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:21.034060 3594 raft_consensus.cc:729] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:21.034298 3594 consensus_queue.cc:260] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:21.034920 3594 ts_tablet_manager.cc:1398] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:21.035531 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-bb1ec320c4114eec8915b7f2bcffc2c3 received from {username='slave'} at 127.20.240.67:54091
I20240503 14:14:21.035640 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-bb1ec320c4114eec8915b7f2bcffc2c3 on tablet bb1ec320c4114eec8915b7f2bcffc2c3 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:21.092900 28171 consensus_queue.cc:237] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
W20240503 14:14:21.095523 28110 consensus_peers.cc:485] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcb14382755441afb4cc5c103425331c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:21.095715 31069 raft_consensus.cc:1261] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:21.095925 30940 raft_consensus.cc:1261] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:21.096354 3697 consensus_queue.cc:1035] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:21.097455 3697 consensus_queue.cc:1035] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:21.099987 3572 raft_consensus.cc:2932] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.102496 30940 raft_consensus.cc:2932] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.103699 31069 raft_consensus.cc:2932] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.113565 27901 catalog_manager.cc:5484] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.127858 30809 raft_consensus.cc:1203] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:21.154781 3758 ts_tablet_manager.cc:902] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:21.160305 30809 consensus_queue.cc:237] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:21.163561 31069 raft_consensus.cc:1261] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:21.163792 28434 raft_consensus.cc:1261] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:21.165201 2991 consensus_queue.cc:1035] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:21.165491 3515 consensus_queue.cc:1035] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:21.166538 30744 consensus_peers.cc:485] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1058a6f8db004b50a51a4c06152c7756. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:21.171608 2991 raft_consensus.cc:2932] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:21.174911 28432 raft_consensus.cc:2932] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:21.175954 3758 tablet_copy_client.cc:308] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:21.176605 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet b96b54a566db4bd1a31e8fb30096c942 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:55553)
I20240503 14:14:21.176733 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet b96b54a566db4bd1a31e8fb30096c942 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:55553: session id = 793a0d5dd1634c5faaf0e1e4851e8294-b96b54a566db4bd1a31e8fb30096c942
I20240503 14:14:21.179500 28454 tablet_copy_source_session.cc:205] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.185693 3758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b96b54a566db4bd1a31e8fb30096c942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.188859 27895 catalog_manager.cc:5484] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:21.193465 31069 raft_consensus.cc:2932] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:21.195318 3758 tablet_copy_client.cc:783] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.195612 3758 tablet_copy_client.cc:647] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.203922 3758 tablet_copy_client.cc:523] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.205745 28302 raft_consensus.cc:1203] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.208181 3758 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:14:21.218624 3758 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.219184 3758 tablet_bootstrap.cc:492] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:14:21.219507 3758 ts_tablet_manager.cc:1367] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 14:14:21.220420 3758 raft_consensus.cc:348] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:21.220724 3758 raft_consensus.cc:729] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:14:21.221036 3758 consensus_queue.cc:260] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:21.221971 3758 ts_tablet_manager.cc:1398] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:21.222836 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-b96b54a566db4bd1a31e8fb30096c942 received from {username='slave'} at 127.20.240.65:55553
I20240503 14:14:21.222951 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-b96b54a566db4bd1a31e8fb30096c942 on tablet b96b54a566db4bd1a31e8fb30096c942 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:14:21.225764 28434 raft_consensus.cc:1203] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.280850 28041 consensus_queue.cc:237] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:21.281040 31069 raft_consensus.cc:1203] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:21.292214 28171 raft_consensus.cc:1261] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:21.292524 31069 raft_consensus.cc:1261] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:21.292860 3475 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:21.293128 27979 consensus_peers.cc:485] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81973d2082ab4bacb61722e51c146f30. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:21.293442 3475 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:21.296097 3475 raft_consensus.cc:2932] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:21.297242 28171 raft_consensus.cc:2932] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:21.301466 31069 raft_consensus.cc:2932] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:21.304885 27895 catalog_manager.cc:5484] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:21.345958 28303 consensus_queue.cc:237] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:21.348848 28432 raft_consensus.cc:1261] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:21.349018 28237 consensus_peers.cc:485] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 609db8c3f3b245eaa31c82915b61188f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:21.349387 2559 consensus_queue.cc:1035] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:21.350412 28171 raft_consensus.cc:1261] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:21.351394 2559 consensus_queue.cc:1035] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:21.367719 2559 raft_consensus.cc:2932] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.370862 28172 raft_consensus.cc:2932] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.375149 28432 raft_consensus.cc:2932] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.376993 27902 catalog_manager.cc:5484] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:21.386556 30941 raft_consensus.cc:1203] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.403033 3815 ts_tablet_manager.cc:902] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:21.416268 3815 tablet_copy_client.cc:308] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:21.417065 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 69d4d86cb68646bc9141c1e6df964561 from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:33841)
I20240503 14:14:21.417232 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 69d4d86cb68646bc9141c1e6df964561 from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:33841: session id = d0dbfb0e43f948c3922926aacc365fc1-69d4d86cb68646bc9141c1e6df964561
I20240503 14:14:21.420698 28454 tablet_copy_source_session.cc:205] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.421949 3815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d4d86cb68646bc9141c1e6df964561. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.428571 3815 tablet_copy_client.cc:783] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.428790 3815 tablet_copy_client.cc:647] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.430646 3815 tablet_copy_client.cc:523] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.433274 3815 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:21.450359 30808 consensus_queue.cc:237] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:21.452714 31069 raft_consensus.cc:1261] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:21.452797 30746 consensus_peers.cc:485] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58ff895fc8e4ddb84e2c1db165799d1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:21.453189 30941 raft_consensus.cc:1261] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:21.453339 3768 consensus_queue.cc:1035] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:21.453735 3762 consensus_queue.cc:1035] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:21.459329 3768 raft_consensus.cc:2932] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.460120 31069 raft_consensus.cc:2932] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.464504 30941 raft_consensus.cc:2932] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.465819 27895 catalog_manager.cc:5484] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:21.478605 3835 ts_tablet_manager.cc:902] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:21.485076 3815 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.486793 3815 tablet_bootstrap.cc:492] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:21.487015 3815 ts_tablet_manager.cc:1367] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.054s user 0.040s sys 0.005s
I20240503 14:14:21.487654 3815 raft_consensus.cc:348] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:21.487854 3815 raft_consensus.cc:729] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:21.488086 3815 consensus_queue.cc:260] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:21.488692 3815 ts_tablet_manager.cc:1398] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:21.489404 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-69d4d86cb68646bc9141c1e6df964561 received from {username='slave'} at 127.20.240.69:33841
I20240503 14:14:21.489500 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-69d4d86cb68646bc9141c1e6df964561 on tablet 69d4d86cb68646bc9141c1e6df964561 with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:21.493031 3835 tablet_copy_client.cc:308] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:21.493924 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 8377b1aa704c43b78f310e7440f6488e from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:37133)
I20240503 14:14:21.494074 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 8377b1aa704c43b78f310e7440f6488e from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:37133: session id = 5388902973134958a030fd63ab4faba4-8377b1aa704c43b78f310e7440f6488e
I20240503 14:14:21.496989 28324 tablet_copy_source_session.cc:205] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.498430 3835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8377b1aa704c43b78f310e7440f6488e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.507278 3835 tablet_copy_client.cc:783] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.507591 3835 tablet_copy_client.cc:647] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.510875 3835 tablet_copy_client.cc:523] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.517782 3835 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:21.527276 3835 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.527946 3835 tablet_bootstrap.cc:492] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:21.528290 3835 ts_tablet_manager.cc:1367] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240503 14:14:21.529520 3835 raft_consensus.cc:348] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:21.529979 3835 raft_consensus.cc:729] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:21.530349 3835 consensus_queue.cc:260] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:21.531661 3835 ts_tablet_manager.cc:1398] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:21.533061 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-8377b1aa704c43b78f310e7440f6488e received from {username='slave'} at 127.20.240.68:37133
I20240503 14:14:21.533196 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session 5388902973134958a030fd63ab4faba4-8377b1aa704c43b78f310e7440f6488e on tablet 8377b1aa704c43b78f310e7440f6488e with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:21.534906 28303 raft_consensus.cc:1203] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.594611 3856 ts_tablet_manager.cc:902] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:14:21.599092 3856 tablet_copy_client.cc:308] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:14:21.599921 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet 1058a6f8db004b50a51a4c06152c7756 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:50623)
I20240503 14:14:21.600064 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet 1058a6f8db004b50a51a4c06152c7756 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:50623: session id = 530893a961af4b95b6b687fa5ed22f9d-1058a6f8db004b50a51a4c06152c7756
I20240503 14:14:21.602717 30830 tablet_copy_source_session.cc:205] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.603981 3856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1058a6f8db004b50a51a4c06152c7756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.610417 3856 tablet_copy_client.cc:783] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.610656 3856 tablet_copy_client.cc:647] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.612696 3856 tablet_copy_client.cc:523] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.616591 3856 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:14:21.637109 3856 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.637710 3856 tablet_bootstrap.cc:492] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:14:21.638068 3856 ts_tablet_manager.cc:1367] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.004s
I20240503 14:14:21.639142 3856 raft_consensus.cc:348] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:21.639516 3856 raft_consensus.cc:729] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:14:21.640050 3869 ts_tablet_manager.cc:902] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:21.639858 3856 consensus_queue.cc:260] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:21.640921 3856 ts_tablet_manager.cc:1398] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:21.642571 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-1058a6f8db004b50a51a4c06152c7756 received from {username='slave'} at 127.20.240.72:50623
I20240503 14:14:21.642693 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-1058a6f8db004b50a51a4c06152c7756 on tablet 1058a6f8db004b50a51a4c06152c7756 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:14:21.691114 3869 tablet_copy_client.cc:308] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:21.692045 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet dcb14382755441afb4cc5c103425331c from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:53253)
I20240503 14:14:21.692214 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet dcb14382755441afb4cc5c103425331c from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:53253: session id = a4cdf05256a0407c8929ac43eb55d44f-dcb14382755441afb4cc5c103425331c
I20240503 14:14:21.695053 28194 tablet_copy_source_session.cc:205] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.696353 3869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb14382755441afb4cc5c103425331c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.698705 28042 raft_consensus.cc:1203] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:21.702762 3869 tablet_copy_client.cc:783] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.702980 3869 tablet_copy_client.cc:647] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.704454 3869 tablet_copy_client.cc:523] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.707644 3869 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:21.716723 3835 ts_tablet_manager.cc:902] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:21.729861 3869 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.730221 3869 tablet_bootstrap.cc:492] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:21.730510 3869 ts_tablet_manager.cc:1367] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.005s
I20240503 14:14:21.731189 3869 raft_consensus.cc:348] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:21.731856 3869 raft_consensus.cc:729] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:21.732059 3869 consensus_queue.cc:260] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:21.732775 3869 ts_tablet_manager.cc:1398] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:14:21.733454 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-dcb14382755441afb4cc5c103425331c received from {username='slave'} at 127.20.240.67:53253
I20240503 14:14:21.733568 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-dcb14382755441afb4cc5c103425331c on tablet dcb14382755441afb4cc5c103425331c with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:21.740782 3835 tablet_copy_client.cc:308] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:21.742322 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 81973d2082ab4bacb61722e51c146f30 from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:35861)
I20240503 14:14:21.742484 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 81973d2082ab4bacb61722e51c146f30 from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:35861: session id = 5388902973134958a030fd63ab4faba4-81973d2082ab4bacb61722e51c146f30
I20240503 14:14:21.745517 28063 tablet_copy_source_session.cc:205] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.747511 3835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81973d2082ab4bacb61722e51c146f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.756603 3835 tablet_copy_client.cc:783] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.756934 3835 tablet_copy_client.cc:647] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:21.774197 3835 tablet_copy_client.cc:523] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:21.779973 3835 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:21.788200 3835 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:21.788619 3835 tablet_bootstrap.cc:492] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:21.788852 3835 ts_tablet_manager.cc:1367] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240503 14:14:21.789567 3835 raft_consensus.cc:348] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:21.789773 3835 raft_consensus.cc:729] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:21.790019 3835 consensus_queue.cc:260] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:21.792642 3835 ts_tablet_manager.cc:1398] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:21.793227 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-81973d2082ab4bacb61722e51c146f30 received from {username='slave'} at 127.20.240.68:35861
I20240503 14:14:21.793359 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 5388902973134958a030fd63ab4faba4-81973d2082ab4bacb61722e51c146f30 on tablet 81973d2082ab4bacb61722e51c146f30 with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:21.885782 30808 raft_consensus.cc:1203] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:21.895392 3815 ts_tablet_manager.cc:902] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:21.900988 3869 ts_tablet_manager.cc:902] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:14:21.981210 3815 tablet_copy_client.cc:308] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:21.982220 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 609db8c3f3b245eaa31c82915b61188f from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:53837)
I20240503 14:14:21.982398 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 609db8c3f3b245eaa31c82915b61188f from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:53837: session id = d0dbfb0e43f948c3922926aacc365fc1-609db8c3f3b245eaa31c82915b61188f
I20240503 14:14:21.984997 28324 tablet_copy_source_session.cc:205] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:21.987171 3815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 609db8c3f3b245eaa31c82915b61188f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:21.997645 3815 tablet_copy_client.cc:783] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:21.997953 3815 tablet_copy_client.cc:647] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:22.000079 3815 tablet_copy_client.cc:523] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:22.003104 3815 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:22.005311 3869 tablet_copy_client.cc:308] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:14:22.006275 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet d58ff895fc8e4ddb84e2c1db165799d1 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:45597)
I20240503 14:14:22.006410 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet d58ff895fc8e4ddb84e2c1db165799d1 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:45597: session id = a4cdf05256a0407c8929ac43eb55d44f-d58ff895fc8e4ddb84e2c1db165799d1
I20240503 14:14:22.009289 30830 tablet_copy_source_session.cc:205] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:22.012979 3869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58ff895fc8e4ddb84e2c1db165799d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:22.023903 3869 tablet_copy_client.cc:783] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:22.024183 3869 tablet_copy_client.cc:647] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:22.025732 28302 raft_consensus.cc:1203] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:22.027375 3869 tablet_copy_client.cc:523] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:22.031363 3869 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:22.039355 28432 raft_consensus.cc:1203] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:22.041149 3869 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:22.041538 3869 tablet_bootstrap.cc:492] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:22.041808 3869 ts_tablet_manager.cc:1367] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 14:14:22.042959 3869 raft_consensus.cc:348] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:22.043247 3869 raft_consensus.cc:729] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.043512 3869 consensus_queue.cc:260] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:22.044139 3869 ts_tablet_manager.cc:1398] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:14:22.045051 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-d58ff895fc8e4ddb84e2c1db165799d1 received from {username='slave'} at 127.20.240.67:45597
I20240503 14:14:22.045154 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-d58ff895fc8e4ddb84e2c1db165799d1 on tablet d58ff895fc8e4ddb84e2c1db165799d1 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:22.079404 3815 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:22.079929 3815 tablet_bootstrap.cc:492] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:22.080224 3815 ts_tablet_manager.cc:1367] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.077s user 0.061s sys 0.009s
I20240503 14:14:22.080930 3815 raft_consensus.cc:348] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:22.081138 3815 raft_consensus.cc:729] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:22.081341 3815 consensus_queue.cc:260] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:22.083422 3815 ts_tablet_manager.cc:1398] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:14:22.084786 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-609db8c3f3b245eaa31c82915b61188f received from {username='slave'} at 127.20.240.69:53837
I20240503 14:14:22.084892 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-609db8c3f3b245eaa31c82915b61188f on tablet 609db8c3f3b245eaa31c82915b61188f with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:22.127540 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:22.163933 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:22.176092 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:22.199745 31199 raft_consensus.cc:1203] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:22.202510 28432 raft_consensus.cc:1203] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:22.222358 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:22.229161 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:22.279621 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:22.279814 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:22.318526 30809 raft_consensus.cc:1203] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:22.330948 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:22.349982 28303 raft_consensus.cc:1203] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:22.750981 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:22.755817 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:22.758468 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:22.782217 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:22.806331 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:22.835647 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:22.841521 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:22.841923 31050 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:14:23.000741 28041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30"
dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59564
I20240503 14:14:23.000996 28041 raft_consensus.cc:595] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:23.079289 3475 raft_consensus.cc:979] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: : Instructing follower fde134d0fa2b4900b9badd2e08fea09d to start an election
I20240503 14:14:23.079452 3475 raft_consensus.cc:1067] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Signalling peer fde134d0fa2b4900b9badd2e08fea09d to start an election
I20240503 14:14:23.080173 31070 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30"
dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
from {username='slave'} at 127.20.240.65:43505
I20240503 14:14:23.080397 31070 raft_consensus.cc:482] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:23.080492 31070 raft_consensus.cc:3037] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:23.083420 31070 raft_consensus.cc:504] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:23.086170 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:14:23.086369 28041 raft_consensus.cc:3032] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:23.086439 28041 raft_consensus.cc:729] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:14:23.086413 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:14:23.086591 28172 raft_consensus.cc:3037] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:23.086599 31070 leader_election.cc:288] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:23.086735 28041 consensus_queue.cc:260] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:23.087123 28041 raft_consensus.cc:3037] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:23.087146 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81973d2082ab4bacb61722e51c146f30" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:14:23.087323 28434 raft_consensus.cc:3037] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:23.089497 28172 raft_consensus.cc:2454] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:23.090021 28434 raft_consensus.cc:2454] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:23.090493 31008 leader_election.cc:302] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:14:23.090543 28041 raft_consensus.cc:2454] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:23.090759 3142 raft_consensus.cc:2781] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:23.090909 3142 raft_consensus.cc:686] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:14:23.091158 3142 consensus_queue.cc:237] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:23.105706 27898 catalog_manager.cc:5484] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 1 to 2, leader changed from 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:23.336616 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:23.370559 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:23.377214 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 14:14:23.429555 27977 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 57926.8us
I20240503 14:14:23.439566 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:23.465971 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:23.475677 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:23.502537 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:23.559301 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:23.577147 28041 raft_consensus.cc:1261] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:23.579648 3918 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:23.627621 28172 raft_consensus.cc:1261] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:23.629657 3918 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:23.681592 28432 raft_consensus.cc:1261] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:23.683020 4083 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
W20240503 14:14:24.011643 31136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 67597.5us
I20240503 14:14:24.081714 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:24.107143 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:24.117601 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:24.151060 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:24.162434 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:24.189778 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:24.282763 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:24.294276 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:24.457818 28432 consensus_queue.cc:237] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:24.459843 30808 raft_consensus.cc:1261] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:24.460120 28302 raft_consensus.cc:1261] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:24.460351 4152 consensus_queue.cc:1035] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:24.460613 4155 consensus_queue.cc:1035] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:24.463307 4154 raft_consensus.cc:2932] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:24.463960 30808 raft_consensus.cc:2932] T 69d4d86cb68646bc9141c1e6df964561 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:24.466653 28302 raft_consensus.cc:2932] T 69d4d86cb68646bc9141c1e6df964561 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:24.469545 27901 catalog_manager.cc:5484] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:24.479996 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d4d86cb68646bc9141c1e6df964561 with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:24.484390 4181 tablet_replica.cc:330] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:24.488232 4181 raft_consensus.cc:2229] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:24.488718 4181 raft_consensus.cc:2258] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:24.489849 4181 ts_tablet_manager.cc:1875] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:24.497970 4181 ts_tablet_manager.cc:1888] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:24.498070 4181 log.cc:1196] T 69d4d86cb68646bc9141c1e6df964561 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/69d4d86cb68646bc9141c1e6df964561
I20240503 14:14:24.498747 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 69d4d86cb68646bc9141c1e6df964561 (table rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]) successfully deleted
I20240503 14:14:24.884472 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:24.890041 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:24.916529 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:24.937938 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:24.946622 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:24.948174 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:25.000455 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:25.029183 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:25.161136 30808 consensus_queue.cc:237] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:25.164232 28432 raft_consensus.cc:1261] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:25.164502 31199 raft_consensus.cc:1261] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:25.164773 3770 consensus_queue.cc:1035] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:25.165745 3770 consensus_queue.cc:1035] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:25.168520 3770 raft_consensus.cc:2932] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:25.170917 28432 raft_consensus.cc:2932] T 1058a6f8db004b50a51a4c06152c7756 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:25.179809 31199 raft_consensus.cc:2932] T 1058a6f8db004b50a51a4c06152c7756 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:25.181289 27899 catalog_manager.cc:5484] T 1058a6f8db004b50a51a4c06152c7756 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:25.200313 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 1058a6f8db004b50a51a4c06152c7756 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:25.202777 4280 tablet_replica.cc:330] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:25.203104 4280 raft_consensus.cc:2229] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:25.203357 4280 raft_consensus.cc:2258] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:25.203989 4280 ts_tablet_manager.cc:1875] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:25.209960 4280 ts_tablet_manager.cc:1888] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:25.210167 4280 log.cc:1196] T 1058a6f8db004b50a51a4c06152c7756 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/1058a6f8db004b50a51a4c06152c7756
I20240503 14:14:25.211334 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 1058a6f8db004b50a51a4c06152c7756 (table rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]) successfully deleted
I20240503 14:14:25.554057 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:25.575747 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:25.577129 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:25.600601 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:25.636600 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:25.712666 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:25.734642 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:25.764389 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:25.938993 28302 consensus_queue.cc:237] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:25.940825 28432 raft_consensus.cc:1261] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:25.940894 30941 raft_consensus.cc:1261] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:25.941373 4219 consensus_queue.cc:1035] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:25.941635 3913 consensus_queue.cc:1035] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:25.944557 4219 raft_consensus.cc:2932] T 8377b1aa704c43b78f310e7440f6488e P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:25.945223 28432 raft_consensus.cc:2932] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:25.948390 30941 raft_consensus.cc:2932] T 8377b1aa704c43b78f310e7440f6488e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:25.953691 27898 catalog_manager.cc:5484] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:25.964593 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 8377b1aa704c43b78f310e7440f6488e with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:25.966231 4366 tablet_replica.cc:330] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:25.966487 4366 raft_consensus.cc:2229] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:25.966684 4366 raft_consensus.cc:2258] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:25.967358 4366 ts_tablet_manager.cc:1875] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:25.972822 4366 ts_tablet_manager.cc:1888] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:25.972976 4366 log.cc:1196] T 8377b1aa704c43b78f310e7440f6488e P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/8377b1aa704c43b78f310e7440f6488e
I20240503 14:14:25.973906 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 8377b1aa704c43b78f310e7440f6488e (table rebalancer_extra_table_44 [id=3744f81450ee4674a77d5671540dc86f]) successfully deleted
I20240503 14:14:26.306102 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:26.327476 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:26.338657 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:26.364547 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:26.408056 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:26.418650 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:26.509913 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:26.533020 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:26.687996 28432 consensus_queue.cc:237] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:26.690017 28043 raft_consensus.cc:1261] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:26.690192 30941 raft_consensus.cc:1261] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:26.690779 4253 consensus_queue.cc:1035] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:26.691182 4253 consensus_queue.cc:1035] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:26.693939 4155 raft_consensus.cc:2932] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:26.697185 28043 raft_consensus.cc:2932] T b96b54a566db4bd1a31e8fb30096c942 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:26.697607 30941 raft_consensus.cc:2932] T b96b54a566db4bd1a31e8fb30096c942 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:26.702240 27896 catalog_manager.cc:5484] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:26.713246 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet b96b54a566db4bd1a31e8fb30096c942 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:26.713778 4461 tablet_replica.cc:330] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:26.713955 4461 raft_consensus.cc:2229] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:26.714556 4461 raft_consensus.cc:2258] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:26.715183 4461 ts_tablet_manager.cc:1875] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:26.719456 4461 ts_tablet_manager.cc:1888] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:26.719542 4461 log.cc:1196] T b96b54a566db4bd1a31e8fb30096c942 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/b96b54a566db4bd1a31e8fb30096c942
I20240503 14:14:26.720094 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet b96b54a566db4bd1a31e8fb30096c942 (table rebalancer_extra_table_56 [id=818c903140564f39abecc41dce78f413]) successfully deleted
I20240503 14:14:27.079770 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:27.080231 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:27.121587 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:27.128652 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 14:14:27.142529 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 67734.3us
I20240503 14:14:27.165515 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:27.184072 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:27.239439 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:27.243204 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:27.423440 28172 consensus_queue.cc:237] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:27.425566 28302 raft_consensus.cc:1261] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:27.426071 31071 raft_consensus.cc:1261] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:27.426108 4430 consensus_queue.cc:1035] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:27.426604 4430 consensus_queue.cc:1035] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:27.429203 4005 raft_consensus.cc:2932] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:27.429843 28302 raft_consensus.cc:2932] T dcb14382755441afb4cc5c103425331c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:27.430191 31071 raft_consensus.cc:2932] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:27.438524 27894 catalog_manager.cc:5484] T dcb14382755441afb4cc5c103425331c P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:27.446317 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb14382755441afb4cc5c103425331c with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:14:27.446871 4550 tablet_replica.cc:330] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:14:27.447732 4550 raft_consensus.cc:2229] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:27.447933 4550 raft_consensus.cc:2258] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:27.448513 4550 ts_tablet_manager.cc:1875] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:27.455371 4550 ts_tablet_manager.cc:1888] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:27.455490 4550 log.cc:1196] T dcb14382755441afb4cc5c103425331c P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/dcb14382755441afb4cc5c103425331c
I20240503 14:14:27.456393 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet dcb14382755441afb4cc5c103425331c (table rebalancer_extra_table_48 [id=6ff1f5f0b64a4e9ab82294397000ba25]) successfully deleted
I20240503 14:14:27.770938 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:27.771198 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:27.778844 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:27.794196 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:27.796607 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:27.807497 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:27.813714 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:27.824539 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:28.023654 30940 consensus_queue.cc:237] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:28.029152 28302 raft_consensus.cc:1261] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:28.029198 31070 raft_consensus.cc:1261] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:28.030053 4398 consensus_queue.cc:1035] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:28.030735 4520 consensus_queue.cc:1035] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:28.033221 4398 raft_consensus.cc:2932] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.034318 31070 raft_consensus.cc:2932] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.039014 28302 raft_consensus.cc:2932] T bb1ec320c4114eec8915b7f2bcffc2c3 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.042606 27896 catalog_manager.cc:5484] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.052948 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet bb1ec320c4114eec8915b7f2bcffc2c3 with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:28.054790 4631 tablet_replica.cc:330] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:28.055085 4631 raft_consensus.cc:2229] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:28.055389 4631 raft_consensus.cc:2258] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:28.056181 4631 ts_tablet_manager.cc:1875] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:28.063151 4631 ts_tablet_manager.cc:1888] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:28.063279 4631 log.cc:1196] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb1ec320c4114eec8915b7f2bcffc2c3
I20240503 14:14:28.064014 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet bb1ec320c4114eec8915b7f2bcffc2c3 (table rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]) successfully deleted
I20240503 14:14:28.340116 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:28.362419 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:28.380031 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:28.397225 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:28.411105 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:28.444996 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:28.495565 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:28.505587 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:28.657502 30808 consensus_queue.cc:237] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:28.667379 28302 raft_consensus.cc:1261] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:28.668089 31071 raft_consensus.cc:1261] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:28.669528 4318 consensus_queue.cc:1035] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:28.669852 4499 consensus_queue.cc:1035] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:28.673501 4318 raft_consensus.cc:2932] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.673907 28302 raft_consensus.cc:2932] T d58ff895fc8e4ddb84e2c1db165799d1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.682337 27895 catalog_manager.cc:5484] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.695580 31071 raft_consensus.cc:2932] T d58ff895fc8e4ddb84e2c1db165799d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:28.706861 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet d58ff895fc8e4ddb84e2c1db165799d1 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:14:28.712189 4715 tablet_replica.cc:330] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:14:28.712524 4715 raft_consensus.cc:2229] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:28.712807 4715 raft_consensus.cc:2258] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:28.714854 4715 ts_tablet_manager.cc:1875] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:28.722386 4715 ts_tablet_manager.cc:1888] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:28.722537 4715 log.cc:1196] T d58ff895fc8e4ddb84e2c1db165799d1 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/d58ff895fc8e4ddb84e2c1db165799d1
I20240503 14:14:28.733108 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet d58ff895fc8e4ddb84e2c1db165799d1 (table rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]) successfully deleted
I20240503 14:14:29.005553 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:29.016022 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:29.027498 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:29.052433 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:29.097131 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:29.115226 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:29.123505 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 14:14:29.213425 28238 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 85572.5us
I20240503 14:14:29.224920 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:29.329890 31070 consensus_queue.cc:237] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:29.331753 30940 raft_consensus.cc:1261] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:29.332049 28173 raft_consensus.cc:1261] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:29.332435 4082 consensus_queue.cc:1035] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:29.332846 3918 consensus_queue.cc:1035] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:29.336035 3918 raft_consensus.cc:2932] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:29.336818 30940 raft_consensus.cc:2932] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:29.339481 28173 raft_consensus.cc:2932] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:29.347321 27893 catalog_manager.cc:5484] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:29.401129 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:14:29.407300 4805 tablet_replica.cc:330] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:14:29.415684 4805 raft_consensus.cc:2229] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:29.415987 4805 raft_consensus.cc:2258] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:29.416652 4805 ts_tablet_manager.cc:1875] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:29.426467 4805 ts_tablet_manager.cc:1888] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:29.426623 4805 log.cc:1196] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/2e1fbc1d1bb6439c9f8b64364ad9ab4b
I20240503 14:14:29.427381 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b (table rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]) successfully deleted
I20240503 14:14:29.756803 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:29.757102 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:29.779883 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:29.787731 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:29.793900 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:29.798467 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:29.817557 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:29.820040 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 14:14:29.908416 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 53633.1us
I20240503 14:14:30.068595 28173 consensus_queue.cc:237] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:30.075246 28432 raft_consensus.cc:1261] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:30.075915 28042 raft_consensus.cc:1261] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:30.075950 4005 consensus_queue.cc:1035] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:30.076782 4005 consensus_queue.cc:1035] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:30.083542 4430 raft_consensus.cc:2932] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:30.087419 28432 raft_consensus.cc:2932] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:30.089918 28042 raft_consensus.cc:2932] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:30.102386 27898 catalog_manager.cc:5484] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:30.117218 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet da7bb0a1956a452cbc1cbbb3999fd7e2 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:30.118607 4889 tablet_replica.cc:330] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:30.118873 4889 raft_consensus.cc:2229] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:30.119227 4889 raft_consensus.cc:2258] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:30.119926 4889 ts_tablet_manager.cc:1875] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:30.124594 4889 ts_tablet_manager.cc:1888] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:30.124708 4889 log.cc:1196] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/da7bb0a1956a452cbc1cbbb3999fd7e2
I20240503 14:14:30.125332 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet da7bb0a1956a452cbc1cbbb3999fd7e2 (table rebalancer_extra_table_62 [id=775fde11483f43fb9c25ad7055d6353f]) successfully deleted
I20240503 14:14:30.441349 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:30.451572 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:30.461398 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:30.462903 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:30.465598 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:30.485795 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:30.493115 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:30.571498 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:30.713733 28434 consensus_queue.cc:237] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:30.715746 31069 raft_consensus.cc:1261] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:30.715763 28302 raft_consensus.cc:1261] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:30.716353 4453 consensus_queue.cc:1035] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:30.716567 4253 consensus_queue.cc:1035] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:30.719139 4253 raft_consensus.cc:2932] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:30.719568 31069 raft_consensus.cc:2932] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:30.720293 28302 raft_consensus.cc:2932] T 48266bf4ad9b4976b639553490ffebbe P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:30.726641 27898 catalog_manager.cc:5484] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:30.739723 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 48266bf4ad9b4976b639553490ffebbe with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:30.740909 4974 tablet_replica.cc:330] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:30.741124 4974 raft_consensus.cc:2229] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:30.741431 4974 raft_consensus.cc:2258] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:30.742131 4974 ts_tablet_manager.cc:1875] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:30.750530 4974 ts_tablet_manager.cc:1888] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:30.750644 4974 log.cc:1196] T 48266bf4ad9b4976b639553490ffebbe P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/48266bf4ad9b4976b639553490ffebbe
I20240503 14:14:30.751325 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 48266bf4ad9b4976b639553490ffebbe (table rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]) successfully deleted
I20240503 14:14:31.048885 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:31.058724 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:31.069347 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:31.128696 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:31.159139 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:31.160753 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:31.171483 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240503 14:14:31.242532 28109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 89339.7us
I20240503 14:14:31.248936 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:31.349450 28302 consensus_queue.cc:237] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:31.351778 28173 raft_consensus.cc:1261] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:31.352247 30807 raft_consensus.cc:1261] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:31.352298 5002 consensus_queue.cc:1035] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:31.352813 4863 consensus_queue.cc:1035] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:31.354926 4863 raft_consensus.cc:2932] T 1bb01c0be7da4fe4b9c7c7160df7debc P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:31.355502 28173 raft_consensus.cc:2932] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:31.355939 30807 raft_consensus.cc:2932] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:31.361901 27900 catalog_manager.cc:5484] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:31.370380 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bb01c0be7da4fe4b9c7c7160df7debc with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58086
I20240503 14:14:31.402652 5059 tablet_replica.cc:330] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:14:31.402884 5059 raft_consensus.cc:2229] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:31.407114 5059 raft_consensus.cc:2258] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:31.407959 5059 ts_tablet_manager.cc:1875] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:31.422441 5059 ts_tablet_manager.cc:1888] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:31.422654 5059 log.cc:1196] T 1bb01c0be7da4fe4b9c7c7160df7debc P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/1bb01c0be7da4fe4b9c7c7160df7debc
I20240503 14:14:31.423450 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 1bb01c0be7da4fe4b9c7c7160df7debc (table rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]) successfully deleted
I20240503 14:14:31.729503 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:31.782132 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:31.793680 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:31.804657 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:31.812747 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:31.837756 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:31.838716 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:31.866068 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:32.026211 28303 consensus_queue.cc:237] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:32.028591 28432 raft_consensus.cc:1261] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:32.029201 5002 consensus_queue.cc:1035] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:32.029420 31199 raft_consensus.cc:1261] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:32.030045 5002 consensus_queue.cc:1035] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:32.032136 5002 raft_consensus.cc:2932] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:32.033104 28432 raft_consensus.cc:2932] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:32.038477 31199 raft_consensus.cc:2932] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:32.046772 27896 catalog_manager.cc:5484] T d7b192797b6c40d4853e6ae4583ef3ca P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:32.071307 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet d7b192797b6c40d4853e6ae4583ef3ca with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:32.088160 5151 tablet_replica.cc:330] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:32.088402 5151 raft_consensus.cc:2229] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:32.088650 5151 raft_consensus.cc:2258] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:32.090664 5151 ts_tablet_manager.cc:1875] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:32.097692 5151 ts_tablet_manager.cc:1888] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:32.097851 5151 log.cc:1196] T d7b192797b6c40d4853e6ae4583ef3ca P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/d7b192797b6c40d4853e6ae4583ef3ca
I20240503 14:14:32.098587 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet d7b192797b6c40d4853e6ae4583ef3ca (table rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]) successfully deleted
I20240503 14:14:32.311007 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:32.321316 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:32.385622 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:32.402436 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:32.405750 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:32.433099 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:32.472023 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:14:32.476894 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 74289.3us
I20240503 14:14:32.501610 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:32.711012 31069 consensus_queue.cc:237] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:32.713202 30809 raft_consensus.cc:1261] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:32.713197 28302 raft_consensus.cc:1261] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:32.713737 5105 consensus_queue.cc:1035] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:32.714211 5105 consensus_queue.cc:1035] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:32.716862 5105 raft_consensus.cc:2932] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:32.719643 30809 raft_consensus.cc:2932] T e57c283674c342169940a5903516ec93 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:32.719980 28302 raft_consensus.cc:2932] T e57c283674c342169940a5903516ec93 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:32.726153 27893 catalog_manager.cc:5484] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:32.753031 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet e57c283674c342169940a5903516ec93 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:14:32.753600 5235 tablet_replica.cc:330] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:14:32.753801 5235 raft_consensus.cc:2229] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:32.754065 5235 raft_consensus.cc:2258] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:32.754735 5235 ts_tablet_manager.cc:1875] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:32.761003 5235 ts_tablet_manager.cc:1888] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:32.761108 5235 log.cc:1196] T e57c283674c342169940a5903516ec93 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/e57c283674c342169940a5903516ec93
I20240503 14:14:32.761934 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet e57c283674c342169940a5903516ec93 (table rebalancer_extra_table_18 [id=cc91d67adf304d0b9deaf55df59cd45a]) successfully deleted
I20240503 14:14:33.081890 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:33.106415 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:33.120029 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:33.166016 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:33.212615 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:33.226646 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:33.239387 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:33.266079 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:33.441273 31069 consensus_queue.cc:237] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:33.443218 28302 raft_consensus.cc:1261] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:33.443464 28172 raft_consensus.cc:1261] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:33.443923 4790 consensus_queue.cc:1035] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:33.444162 5105 consensus_queue.cc:1035] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:33.447059 5105 raft_consensus.cc:2932] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:33.450400 28302 raft_consensus.cc:2932] T e571695c819e485483ece09278ee1e09 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:33.450601 28172 raft_consensus.cc:2932] T e571695c819e485483ece09278ee1e09 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:33.456262 27893 catalog_manager.cc:5484] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:33.461984 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet e571695c819e485483ece09278ee1e09 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:14:33.462590 5321 tablet_replica.cc:330] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:14:33.462852 5321 raft_consensus.cc:2229] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:33.463162 5321 raft_consensus.cc:2258] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:33.464002 5321 ts_tablet_manager.cc:1875] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:33.471982 5321 ts_tablet_manager.cc:1888] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:33.472096 5321 log.cc:1196] T e571695c819e485483ece09278ee1e09 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/e571695c819e485483ece09278ee1e09
I20240503 14:14:33.473057 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet e571695c819e485483ece09278ee1e09 (table rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]) successfully deleted
I20240503 14:14:33.745098 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:33.779430 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:33.815341 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:33.834266 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:33.841017 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:33.854609 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:33.856104 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:33.856408 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 14:14:33.889639 31136 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 63279.3us
I20240503 14:14:34.065279 30941 consensus_queue.cc:237] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:34.067283 30808 raft_consensus.cc:1261] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:34.067966 31199 raft_consensus.cc:1261] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:34.068490 5202 consensus_queue.cc:1035] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:34.069001 5202 consensus_queue.cc:1035] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:34.071449 5202 raft_consensus.cc:2932] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.072373 30808 raft_consensus.cc:2932] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.073012 31199 raft_consensus.cc:2932] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.082945 27900 catalog_manager.cc:5484] T 9088c0106bf249cab5b35c1171a7e7a0 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.112416 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9088c0106bf249cab5b35c1171a7e7a0 with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58086
I20240503 14:14:34.116619 5410 tablet_replica.cc:330] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:14:34.116833 5410 raft_consensus.cc:2229] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:34.117026 5410 raft_consensus.cc:2258] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:34.119097 5410 ts_tablet_manager.cc:1875] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:34.130539 5410 ts_tablet_manager.cc:1888] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:34.130709 5410 log.cc:1196] T 9088c0106bf249cab5b35c1171a7e7a0 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/9088c0106bf249cab5b35c1171a7e7a0
I20240503 14:14:34.132006 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 9088c0106bf249cab5b35c1171a7e7a0 (table rebalancer_extra_table_52 [id=548f97bb3bcd4f4bb98698fc00bc1ca5]) successfully deleted
I20240503 14:14:34.194976 28041 consensus_queue.cc:237] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:34.197345 31071 raft_consensus.cc:1261] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:34.198424 27977 consensus_peers.cc:485] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8dd90464f4b84baabd90a44a122e28b1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:34.198827 30808 raft_consensus.cc:1261] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:34.199229 5033 consensus_queue.cc:1035] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:34.199851 4799 consensus_queue.cc:1035] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:34.203073 4799 raft_consensus.cc:2932] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:34.203814 31071 raft_consensus.cc:2932] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:34.209901 27893 catalog_manager.cc:5484] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:34.213860 30808 raft_consensus.cc:2932] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:34.533895 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:34.559337 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:34.595137 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:34.626792 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:34.632524 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:34.634083 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:34.708812 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:34.768899 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:34.769714 5509 ts_tablet_manager.cc:902] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:34.781582 5509 tablet_copy_client.cc:308] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:34.786777 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 8dd90464f4b84baabd90a44a122e28b1 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:32803)
I20240503 14:14:34.786958 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 8dd90464f4b84baabd90a44a122e28b1 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:32803: session id = 530893a961af4b95b6b687fa5ed22f9d-8dd90464f4b84baabd90a44a122e28b1
I20240503 14:14:34.789593 28063 tablet_copy_source_session.cc:205] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:34.792438 5509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8dd90464f4b84baabd90a44a122e28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:34.799378 5509 tablet_copy_client.cc:783] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:34.800107 5509 tablet_copy_client.cc:647] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:34.802052 5509 tablet_copy_client.cc:523] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:34.806879 5509 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:14:34.815559 5509 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:34.815915 5509 tablet_bootstrap.cc:492] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:14:34.816160 5509 ts_tablet_manager.cc:1367] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240503 14:14:34.816713 5509 raft_consensus.cc:348] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:34.816877 5509 raft_consensus.cc:729] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:14:34.817104 5509 consensus_queue.cc:260] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:34.817574 5509 ts_tablet_manager.cc:1398] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:34.818217 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-8dd90464f4b84baabd90a44a122e28b1 received from {username='slave'} at 127.20.240.72:32803
I20240503 14:14:34.818354 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-8dd90464f4b84baabd90a44a122e28b1 on tablet 8dd90464f4b84baabd90a44a122e28b1 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:14:34.866114 28302 consensus_queue.cc:237] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:34.868083 28432 raft_consensus.cc:1261] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:34.868548 30807 raft_consensus.cc:1261] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:34.868677 5270 consensus_queue.cc:1035] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:34.870092 5270 consensus_queue.cc:1035] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:34.872710 5270 raft_consensus.cc:2932] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.877765 28432 raft_consensus.cc:2932] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.880401 27897 catalog_manager.cc:5484] T 609db8c3f3b245eaa31c82915b61188f P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.904984 30808 raft_consensus.cc:2932] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:34.953104 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 609db8c3f3b245eaa31c82915b61188f with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:34.953804 5545 tablet_replica.cc:330] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:34.954085 5545 raft_consensus.cc:2229] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:34.954406 5545 raft_consensus.cc:2258] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:34.955447 5545 ts_tablet_manager.cc:1875] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:34.962955 5545 ts_tablet_manager.cc:1888] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:34.963129 5545 log.cc:1196] T 609db8c3f3b245eaa31c82915b61188f P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/609db8c3f3b245eaa31c82915b61188f
I20240503 14:14:34.963997 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 609db8c3f3b245eaa31c82915b61188f (table rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]) successfully deleted
I20240503 14:14:35.134966 31203 raft_consensus.cc:1203] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:35.263046 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:35.293128 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:35.301214 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:35.315827 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:35.317281 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:35.317638 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:35.318373 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:35.342816 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:35.565757 31071 consensus_queue.cc:237] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:35.567631 28434 raft_consensus.cc:1261] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:35.567646 28172 raft_consensus.cc:1261] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:35.568104 5315 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:35.568394 4790 consensus_queue.cc:1035] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:35.570878 4790 raft_consensus.cc:2932] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:35.571624 28434 raft_consensus.cc:2932] T 81973d2082ab4bacb61722e51c146f30 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:35.573948 28172 raft_consensus.cc:2932] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:35.582043 27896 catalog_manager.cc:5484] T 81973d2082ab4bacb61722e51c146f30 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:35.587309 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 81973d2082ab4bacb61722e51c146f30 with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52616
I20240503 14:14:35.598592 5635 tablet_replica.cc:330] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:14:35.598826 5635 raft_consensus.cc:2229] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:35.599140 5635 raft_consensus.cc:2258] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:35.599990 5635 ts_tablet_manager.cc:1875] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:35.606366 5635 ts_tablet_manager.cc:1888] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:14:35.606477 5635 log.cc:1196] T 81973d2082ab4bacb61722e51c146f30 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/81973d2082ab4bacb61722e51c146f30
I20240503 14:14:35.607604 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 81973d2082ab4bacb61722e51c146f30 (table rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]) successfully deleted
I20240503 14:14:36.007725 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:36.022367 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:36.040802 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:36.048744 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:36.050758 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:36.054874 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:36.056975 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:36.085536 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:36.294734 28041 consensus_queue.cc:237] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:36.296505 30808 raft_consensus.cc:1261] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:36.296517 31199 raft_consensus.cc:1261] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:36.297175 5442 consensus_queue.cc:1035] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:36.297622 5442 consensus_queue.cc:1035] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:36.299715 5033 raft_consensus.cc:2932] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.315887 27896 catalog_manager.cc:5484] T 8dd90464f4b84baabd90a44a122e28b1 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.316257 30808 raft_consensus.cc:2932] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.325016 31199 raft_consensus.cc:2932] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.337877 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 8dd90464f4b84baabd90a44a122e28b1 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:36.338718 5767 tablet_replica.cc:330] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:36.338966 5767 raft_consensus.cc:2229] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:36.339143 5767 raft_consensus.cc:2258] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:36.339623 5767 ts_tablet_manager.cc:1875] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:36.344794 5767 ts_tablet_manager.cc:1888] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:36.344939 5767 log.cc:1196] T 8dd90464f4b84baabd90a44a122e28b1 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/8dd90464f4b84baabd90a44a122e28b1
I20240503 14:14:36.345628 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 8dd90464f4b84baabd90a44a122e28b1 (table rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]) successfully deleted
I20240503 14:14:36.461232 28173 consensus_queue.cc:237] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:36.478731 30940 raft_consensus.cc:1261] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.479372 5503 consensus_queue.cc:1035] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:36.479566 30808 raft_consensus.cc:1261] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.480175 5503 consensus_queue.cc:1035] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:36.480891 28110 consensus_peers.cc:485] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0d2600fd84a4d839dc3366b08485e42. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.483342 5370 raft_consensus.cc:2932] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:36.483912 30940 raft_consensus.cc:2932] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:36.488162 30808 raft_consensus.cc:2932] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:36.490705 27896 catalog_manager.cc:5484] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:36.550429 28173 consensus_queue.cc:237] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:36.552698 28041 raft_consensus.cc:1261] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:36.552834 28109 consensus_peers.cc:485] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a816d34a63b945b08eec771d9e7c2d55. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.553260 5809 consensus_queue.cc:1035] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.553417 28302 raft_consensus.cc:1261] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.553965 5809 consensus_queue.cc:1035] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.556696 5805 raft_consensus.cc:2932] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.558967 28302 raft_consensus.cc:2932] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.563376 27895 catalog_manager.cc:5484] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.575347 28043 raft_consensus.cc:2932] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.613237 28434 consensus_queue.cc:237] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:36.615355 28303 raft_consensus.cc:1261] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.615828 28173 raft_consensus.cc:1261] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:36.615947 28368 consensus_peers.cc:485] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 664cadf5232f4768adf6438e57f7b7c9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.616581 5505 consensus_queue.cc:1035] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.616933 4253 consensus_queue.cc:1035] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.628283 4253 raft_consensus.cc:2932] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.628844 28173 raft_consensus.cc:2932] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.636270 27895 catalog_manager.cc:5484] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.643457 28303 raft_consensus.cc:2932] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:36.695351 30940 consensus_queue.cc:237] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
W20240503 14:14:36.698112 30877 consensus_peers.cc:485] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f4fbfaadc474d59bfb23107f0342f7c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.698153 28043 raft_consensus.cc:1261] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.698618 30807 raft_consensus.cc:1261] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.698648 5864 consensus_queue.cc:1035] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:36.699136 5295 consensus_queue.cc:1035] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:36.707593 5295 raft_consensus.cc:2932] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:36.708511 28043 raft_consensus.cc:2932] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:36.715119 30807 raft_consensus.cc:2932] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:36.717985 27900 catalog_manager.cc:5484] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:36.755381 28040 consensus_queue.cc:237] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:36.758324 28173 raft_consensus.cc:1261] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.758392 28303 raft_consensus.cc:1261] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.758924 5442 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:36.759326 5757 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:36.760128 27978 consensus_peers.cc:485] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38cf3aab9e1b4f28842ac16d02be2562. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.767405 5442 raft_consensus.cc:2932] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:36.767987 28303 raft_consensus.cc:2932] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:36.771865 28173 raft_consensus.cc:2932] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:36.774631 27900 catalog_manager.cc:5484] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:36.812089 28434 consensus_queue.cc:237] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
W20240503 14:14:36.814229 28369 consensus_peers.cc:485] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac08c6dba4dc4e90ab11a155ec6ecdfd. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.814348 28303 raft_consensus.cc:1261] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.814550 28173 raft_consensus.cc:1261] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:36.814850 4253 consensus_queue.cc:1035] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.815903 5845 consensus_queue.cc:1035] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:36.818084 5845 raft_consensus.cc:2932] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.818703 28173 raft_consensus.cc:2932] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.822402 28303 raft_consensus.cc:2932] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.824759 27896 catalog_manager.cc:5484] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:36.858741 28173 consensus_queue.cc:237] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:36.861444 31070 raft_consensus.cc:1261] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:36.861644 28107 consensus_peers.cc:485] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0abfcd38ecc64acfac35c676b941581a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.861738 30939 raft_consensus.cc:1261] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.862037 5809 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:36.862267 5503 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:36.866837 5809 raft_consensus.cc:2932] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:36.867401 30939 raft_consensus.cc:2932] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:36.870918 31070 raft_consensus.cc:2932] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:36.873420 27895 catalog_manager.cc:5484] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:36.976492 28303 consensus_queue.cc:237] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:36.979177 31199 raft_consensus.cc:1261] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.979274 28040 raft_consensus.cc:1261] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:36.979751 5736 consensus_queue.cc:1035] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:36.980213 28240 consensus_peers.cc:485] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8baf804d83954d288576c313efe3d1cb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:36.980264 5270 consensus_queue.cc:1035] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:36.983089 5941 raft_consensus.cc:2932] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:36.983768 28040 raft_consensus.cc:2932] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:36.991465 27896 catalog_manager.cc:5484] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:36.993199 31199 raft_consensus.cc:2932] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:37.025681 28303 consensus_queue.cc:237] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:37.028010 28040 raft_consensus.cc:1261] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.028187 28173 raft_consensus.cc:1261] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:37.028519 28239 consensus_peers.cc:485] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f133ebe695a74f86be1a37ae661719ec. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.028522 5270 consensus_queue.cc:1035] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.028754 5941 consensus_queue.cc:1035] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.035454 5941 raft_consensus.cc:2932] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.036149 28173 raft_consensus.cc:2932] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.041237 28040 raft_consensus.cc:2932] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.042035 27895 catalog_manager.cc:5484] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.082726 5974 ts_tablet_manager.cc:902] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:37.092765 5976 ts_tablet_manager.cc:902] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:37.105427 5976 tablet_copy_client.cc:308] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:37.105659 31199 consensus_queue.cc:237] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:37.106137 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 1f4fbfaadc474d59bfb23107f0342f7c from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:36635)
I20240503 14:14:37.106364 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 1f4fbfaadc474d59bfb23107f0342f7c from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:36635: session id = 5388902973134958a030fd63ab4faba4-1f4fbfaadc474d59bfb23107f0342f7c
I20240503 14:14:37.109962 30961 tablet_copy_source_session.cc:205] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 14:14:37.111793 31139 consensus_peers.cc:485] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e389fcad67cc4add9db2f34eb57745c8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.112033 5976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4fbfaadc474d59bfb23107f0342f7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.112655 28040 raft_consensus.cc:1261] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:37.112887 31071 raft_consensus.cc:1261] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:37.113385 5763 consensus_queue.cc:1035] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:37.113646 5608 consensus_queue.cc:1035] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:37.114409 5974 tablet_copy_client.cc:308] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:37.115664 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet d0d2600fd84a4d839dc3366b08485e42 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:50673)
I20240503 14:14:37.115789 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet d0d2600fd84a4d839dc3366b08485e42 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:50673: session id = 793a0d5dd1634c5faaf0e1e4851e8294-d0d2600fd84a4d839dc3366b08485e42
I20240503 14:14:37.118418 28194 tablet_copy_source_session.cc:205] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.120795 5976 tablet_copy_client.cc:783] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.121080 5976 tablet_copy_client.cc:647] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.122785 5976 tablet_copy_client.cc:523] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.123203 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d2600fd84a4d839dc3366b08485e42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.127068 5976 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:37.128517 5974 tablet_copy_client.cc:783] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.128721 5974 tablet_copy_client.cc:647] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.130189 5974 tablet_copy_client.cc:523] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.133466 5974 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:14:37.135053 5763 raft_consensus.cc:2932] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.141148 28043 raft_consensus.cc:2932] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.146742 5974 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.147131 5974 tablet_bootstrap.cc:492] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:14:37.147361 5974 ts_tablet_manager.cc:1367] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.005s
I20240503 14:14:37.160554 27896 catalog_manager.cc:5484] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.160789 5974 raft_consensus.cc:348] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:37.160975 5974 raft_consensus.cc:729] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.161147 5974 consensus_queue.cc:260] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:37.161674 5974 ts_tablet_manager.cc:1398] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:14:37.163875 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-d0d2600fd84a4d839dc3366b08485e42 received from {username='slave'} at 127.20.240.65:50673
I20240503 14:14:37.163983 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-d0d2600fd84a4d839dc3366b08485e42 on tablet d0d2600fd84a4d839dc3366b08485e42 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:14:37.168013 31070 raft_consensus.cc:2932] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.187897 5996 ts_tablet_manager.cc:902] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:37.192005 5976 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.192472 5976 tablet_bootstrap.cc:492] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:37.192749 5976 ts_tablet_manager.cc:1367] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.066s user 0.014s sys 0.024s
I20240503 14:14:37.193635 5976 raft_consensus.cc:348] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:37.193918 5976 raft_consensus.cc:729] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.194389 5976 consensus_queue.cc:260] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:37.195041 5976 ts_tablet_manager.cc:1398] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240503 14:14:37.195943 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-1f4fbfaadc474d59bfb23107f0342f7c received from {username='slave'} at 127.20.240.68:36635
I20240503 14:14:37.196130 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 5388902973134958a030fd63ab4faba4-1f4fbfaadc474d59bfb23107f0342f7c on tablet 1f4fbfaadc474d59bfb23107f0342f7c with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:37.199700 5995 ts_tablet_manager.cc:902] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:37.200429 5994 ts_tablet_manager.cc:902] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:37.222306 5996 tablet_copy_client.cc:308] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:37.222419 5995 tablet_copy_client.cc:308] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:37.222986 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 664cadf5232f4768adf6438e57f7b7c9 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:57431)
I20240503 14:14:37.223021 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet a816d34a63b945b08eec771d9e7c2d55 from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:43001)
I20240503 14:14:37.223134 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 664cadf5232f4768adf6438e57f7b7c9 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:57431: session id = 530893a961af4b95b6b687fa5ed22f9d-664cadf5232f4768adf6438e57f7b7c9
I20240503 14:14:37.223193 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet a816d34a63b945b08eec771d9e7c2d55 from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:43001: session id = fde134d0fa2b4900b9badd2e08fea09d-a816d34a63b945b08eec771d9e7c2d55
I20240503 14:14:37.226107 28194 tablet_copy_source_session.cc:205] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.226440 28454 tablet_copy_source_session.cc:205] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.229861 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 664cadf5232f4768adf6438e57f7b7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.232134 5995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a816d34a63b945b08eec771d9e7c2d55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.238730 5995 tablet_copy_client.cc:783] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.238946 5996 tablet_copy_client.cc:783] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.239073 5995 tablet_copy_client.cc:647] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.239179 5996 tablet_copy_client.cc:647] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.242527 5995 tablet_copy_client.cc:523] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.245072 5996 tablet_copy_client.cc:523] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.249547 5995 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:37.250192 5996 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:14:37.274762 6019 ts_tablet_manager.cc:902] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:37.297231 6019 tablet_copy_client.cc:308] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:37.297983 5994 tablet_copy_client.cc:308] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:37.298120 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:60297)
I20240503 14:14:37.298326 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet ac08c6dba4dc4e90ab11a155ec6ecdfd from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:60297: session id = fde134d0fa2b4900b9badd2e08fea09d-ac08c6dba4dc4e90ab11a155ec6ecdfd
I20240503 14:14:37.301348 28454 tablet_copy_source_session.cc:205] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.302244 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 38cf3aab9e1b4f28842ac16d02be2562 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:48859)
I20240503 14:14:37.304865 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 38cf3aab9e1b4f28842ac16d02be2562 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:48859: session id = 6bbc686598f24f518d7ca6bb2063058d-38cf3aab9e1b4f28842ac16d02be2562
I20240503 14:14:37.313335 28063 tablet_copy_source_session.cc:205] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.324144 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38cf3aab9e1b4f28842ac16d02be2562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.324209 6019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.331257 5994 tablet_copy_client.cc:783] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.331705 5994 tablet_copy_client.cc:647] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.334973 5994 tablet_copy_client.cc:523] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.338054 5994 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:37.338824 6019 tablet_copy_client.cc:783] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.339164 6019 tablet_copy_client.cc:647] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.341141 6019 tablet_copy_client.cc:523] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.346376 6019 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:37.360685 5994 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.361111 5994 tablet_bootstrap.cc:492] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:37.361392 5994 ts_tablet_manager.cc:1367] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240503 14:14:37.362133 5994 raft_consensus.cc:348] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:37.362399 5994 raft_consensus.cc:729] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.368450 6035 ts_tablet_manager.cc:902] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:37.368415 5994 consensus_queue.cc:260] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:37.372701 5994 ts_tablet_manager.cc:1398] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:14:37.373751 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-38cf3aab9e1b4f28842ac16d02be2562 received from {username='slave'} at 127.20.240.70:48859
I20240503 14:14:37.373857 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-38cf3aab9e1b4f28842ac16d02be2562 on tablet 38cf3aab9e1b4f28842ac16d02be2562 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:37.382869 28040 consensus_queue.cc:237] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
W20240503 14:14:37.399067 27978 consensus_peers.cc:485] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f25a3bcdef849499e4a02f01f4dbc0e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.399348 28173 raft_consensus.cc:1261] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.399864 6028 consensus_queue.cc:1035] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.403851 28434 raft_consensus.cc:1261] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.405011 5442 consensus_queue.cc:1035] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.411175 5976 ts_tablet_manager.cc:902] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:37.430857 6035 tablet_copy_client.cc:308] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:37.431746 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet 0abfcd38ecc64acfac35c676b941581a from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:47863)
I20240503 14:14:37.431900 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet 0abfcd38ecc64acfac35c676b941581a from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:47863: session id = d0dbfb0e43f948c3922926aacc365fc1-0abfcd38ecc64acfac35c676b941581a
I20240503 14:14:37.434610 28194 tablet_copy_source_session.cc:205] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.434914 5757 raft_consensus.cc:2932] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.436431 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abfcd38ecc64acfac35c676b941581a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.438475 28434 raft_consensus.cc:2932] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.446415 6019 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.447211 6019 tablet_bootstrap.cc:492] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:37.447477 6019 ts_tablet_manager.cc:1367] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.101s user 0.043s sys 0.012s
I20240503 14:14:37.448334 6019 raft_consensus.cc:348] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:37.448557 6019 raft_consensus.cc:729] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.448822 6019 consensus_queue.cc:260] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:37.451364 6019 ts_tablet_manager.cc:1398] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:14:37.452440 5976 tablet_copy_client.cc:308] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:37.453470 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 8baf804d83954d288576c313efe3d1cb from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:37133)
I20240503 14:14:37.453612 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 8baf804d83954d288576c313efe3d1cb from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:37133: session id = 5388902973134958a030fd63ab4faba4-8baf804d83954d288576c313efe3d1cb
I20240503 14:14:37.454193 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-ac08c6dba4dc4e90ab11a155ec6ecdfd received from {username='slave'} at 127.20.240.71:60297
I20240503 14:14:37.454311 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-ac08c6dba4dc4e90ab11a155ec6ecdfd on tablet ac08c6dba4dc4e90ab11a155ec6ecdfd with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:37.455600 28433 raft_consensus.cc:1203] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:37.456395 28324 tablet_copy_source_session.cc:205] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.462504 6035 tablet_copy_client.cc:783] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.462743 6035 tablet_copy_client.cc:647] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.465953 6035 tablet_copy_client.cc:523] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.469311 6035 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:37.469113 28172 raft_consensus.cc:2932] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.476250 27895 catalog_manager.cc:5484] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.483095 6035 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.483464 6035 tablet_bootstrap.cc:492] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:37.483682 6035 ts_tablet_manager.cc:1367] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20240503 14:14:37.484761 6035 raft_consensus.cc:348] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:37.484990 6035 raft_consensus.cc:729] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.485234 6035 consensus_queue.cc:260] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:37.486732 6035 ts_tablet_manager.cc:1398] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:14:37.487516 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-0abfcd38ecc64acfac35c676b941581a received from {username='slave'} at 127.20.240.69:47863
I20240503 14:14:37.487612 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-0abfcd38ecc64acfac35c676b941581a on tablet 0abfcd38ecc64acfac35c676b941581a with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:37.488835 5976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8baf804d83954d288576c313efe3d1cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.500931 5976 tablet_copy_client.cc:783] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.501156 5976 tablet_copy_client.cc:647] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.503985 5976 tablet_copy_client.cc:523] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.507580 5976 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:37.513320 5995 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.513814 5995 tablet_bootstrap.cc:492] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:37.514106 5995 ts_tablet_manager.cc:1367] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.265s user 0.050s sys 0.002s
I20240503 14:14:37.520965 5976 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.521330 5976 tablet_bootstrap.cc:492] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:37.521538 5976 ts_tablet_manager.cc:1367] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240503 14:14:37.522130 5976 raft_consensus.cc:348] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:37.522377 5976 raft_consensus.cc:729] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.522614 5976 consensus_queue.cc:260] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:37.523227 5976 ts_tablet_manager.cc:1398] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:37.524425 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-8baf804d83954d288576c313efe3d1cb received from {username='slave'} at 127.20.240.68:37133
I20240503 14:14:37.524647 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session 5388902973134958a030fd63ab4faba4-8baf804d83954d288576c313efe3d1cb on tablet 8baf804d83954d288576c313efe3d1cb with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:37.529834 5996 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.530243 5996 tablet_bootstrap.cc:492] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:14:37.538579 5996 ts_tablet_manager.cc:1367] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.288s user 0.049s sys 0.014s
I20240503 14:14:37.539251 5996 raft_consensus.cc:348] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:37.539485 5996 raft_consensus.cc:729] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.539705 5996 consensus_queue.cc:260] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:37.540115 5995 raft_consensus.cc:348] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:37.540341 5996 ts_tablet_manager.cc:1398] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:37.540375 5995 raft_consensus.cc:729] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.541239 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-664cadf5232f4768adf6438e57f7b7c9 received from {username='slave'} at 127.20.240.72:57431
I20240503 14:14:37.541332 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-664cadf5232f4768adf6438e57f7b7c9 on tablet 664cadf5232f4768adf6438e57f7b7c9 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:14:37.540629 5995 consensus_queue.cc:260] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:37.542460 5995 ts_tablet_manager.cc:1398] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20240503 14:14:37.546988 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-a816d34a63b945b08eec771d9e7c2d55 received from {username='slave'} at 127.20.240.71:43001
I20240503 14:14:37.547098 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-a816d34a63b945b08eec771d9e7c2d55 on tablet a816d34a63b945b08eec771d9e7c2d55 with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:37.556936 28040 raft_consensus.cc:1203] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:37.591887 5994 ts_tablet_manager.cc:902] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:37.594110 31071 raft_consensus.cc:1203] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:37.599882 31199 raft_consensus.cc:1203] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:37.601996 5994 tablet_copy_client.cc:308] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:37.602627 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet f133ebe695a74f86be1a37ae661719ec from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:57239)
I20240503 14:14:37.602775 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet f133ebe695a74f86be1a37ae661719ec from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:57239: session id = 6bbc686598f24f518d7ca6bb2063058d-f133ebe695a74f86be1a37ae661719ec
I20240503 14:14:37.605275 28324 tablet_copy_source_session.cc:205] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.606417 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f133ebe695a74f86be1a37ae661719ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.613111 5994 tablet_copy_client.cc:783] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.613409 5994 tablet_copy_client.cc:647] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.614977 5994 tablet_copy_client.cc:523] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.616312 31071 consensus_queue.cc:237] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:37.618098 5994 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:37.620527 30808 raft_consensus.cc:1261] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:37.621007 5908 consensus_queue.cc:1035] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:37.621774 31006 consensus_peers.cc:485] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c761d663c0bd48529edc9e454d5292ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.623093 28172 raft_consensus.cc:1261] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:37.623525 5908 consensus_queue.cc:1035] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:37.637351 30940 raft_consensus.cc:1203] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:37.644002 5908 raft_consensus.cc:2932] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:37.652650 28173 raft_consensus.cc:2932] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:37.659690 27895 catalog_manager.cc:5484] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:37.662875 30808 raft_consensus.cc:2932] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:37.673084 6075 ts_tablet_manager.cc:902] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:14:37.674377 5994 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.674863 5994 tablet_bootstrap.cc:492] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:37.675123 5994 ts_tablet_manager.cc:1367] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.057s user 0.043s sys 0.000s
I20240503 14:14:37.675933 5994 raft_consensus.cc:348] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:37.676496 5994 raft_consensus.cc:729] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.676779 5994 consensus_queue.cc:260] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:37.677456 5994 ts_tablet_manager.cc:1398] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:37.678377 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-f133ebe695a74f86be1a37ae661719ec received from {username='slave'} at 127.20.240.70:57239
I20240503 14:14:37.678480 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-f133ebe695a74f86be1a37ae661719ec on tablet f133ebe695a74f86be1a37ae661719ec with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:37.720669 6075 tablet_copy_client.cc:308] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.72:41331
I20240503 14:14:37.721380 31223 tablet_copy_service.cc:140] P 530893a961af4b95b6b687fa5ed22f9d: Received BeginTabletCopySession request for tablet e389fcad67cc4add9db2f34eb57745c8 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:59683)
I20240503 14:14:37.721514 31223 tablet_copy_service.cc:161] P 530893a961af4b95b6b687fa5ed22f9d: Beginning new tablet copy session on tablet e389fcad67cc4add9db2f34eb57745c8 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:59683: session id = a4cdf05256a0407c8929ac43eb55d44f-e389fcad67cc4add9db2f34eb57745c8
I20240503 14:14:37.723942 31223 tablet_copy_source_session.cc:205] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.725240 6075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e389fcad67cc4add9db2f34eb57745c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.731087 6075 tablet_copy_client.cc:783] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.731304 6075 tablet_copy_client.cc:647] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.733714 6075 tablet_copy_client.cc:523] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.737450 6075 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:37.743216 6075 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:37.743510 6075 tablet_bootstrap.cc:492] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:37.743696 6075 ts_tablet_manager.cc:1367] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:14:37.744222 6075 raft_consensus.cc:348] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:37.744379 6075 raft_consensus.cc:729] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:37.744589 6075 consensus_queue.cc:260] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:37.745098 6075 ts_tablet_manager.cc:1398] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:14:37.745743 31223 tablet_copy_service.cc:342] P 530893a961af4b95b6b687fa5ed22f9d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-e389fcad67cc4add9db2f34eb57745c8 received from {username='slave'} at 127.20.240.67:59683
I20240503 14:14:37.745873 31223 tablet_copy_service.cc:433] P 530893a961af4b95b6b687fa5ed22f9d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-e389fcad67cc4add9db2f34eb57745c8 on tablet e389fcad67cc4add9db2f34eb57745c8 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:37.756001 28040 consensus_queue.cc:237] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:37.758356 31071 raft_consensus.cc:1261] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:37.758385 31199 raft_consensus.cc:1261] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:37.758870 5442 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:37.759723 5442 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:37.760655 27979 consensus_peers.cc:485] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7242c3728b84084a738fde7ae32ec0a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.763167 5442 raft_consensus.cc:2932] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:37.763772 31071 raft_consensus.cc:2932] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:37.770267 31199 raft_consensus.cc:2932] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:37.774088 27896 catalog_manager.cc:5484] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:37.818866 30808 consensus_queue.cc:237] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:37.821008 28434 raft_consensus.cc:1261] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:37.821027 30746 consensus_peers.cc:485] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96773dd7242d4223bdc663ee53bfdd3f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.821202 31070 raft_consensus.cc:1261] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:37.821470 6055 consensus_queue.cc:1035] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:37.822289 6055 consensus_queue.cc:1035] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:37.824756 30808 raft_consensus.cc:1203] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:37.825613 6055 raft_consensus.cc:2932] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.826529 31070 raft_consensus.cc:2932] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.830847 28434 raft_consensus.cc:2932] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.834086 27895 catalog_manager.cc:5484] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:37.866369 28040 consensus_queue.cc:237] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:37.868477 28173 raft_consensus.cc:1261] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:37.868659 27978 consensus_peers.cc:485] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb88c79b4b1a40c2b45c2dd79ab894d4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:37.868938 28434 raft_consensus.cc:1261] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:37.869086 6100 consensus_queue.cc:1035] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.869407 6028 consensus_queue.cc:1035] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:37.872045 6100 raft_consensus.cc:2932] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.875080 28434 raft_consensus.cc:2932] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.875665 5994 ts_tablet_manager.cc:902] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:37.876859 28173 raft_consensus.cc:2932] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.882087 31070 raft_consensus.cc:1203] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:37.883541 27895 catalog_manager.cc:5484] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:37.914733 5994 tablet_copy_client.cc:308] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:37.915405 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 5f25a3bcdef849499e4a02f01f4dbc0e from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:48859)
I20240503 14:14:37.915553 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 5f25a3bcdef849499e4a02f01f4dbc0e from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:48859: session id = 6bbc686598f24f518d7ca6bb2063058d-5f25a3bcdef849499e4a02f01f4dbc0e
I20240503 14:14:37.918418 28063 tablet_copy_source_session.cc:205] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:37.919917 28434 raft_consensus.cc:1203] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:37.920075 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f25a3bcdef849499e4a02f01f4dbc0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:37.936861 5994 tablet_copy_client.cc:783] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:37.937230 5994 tablet_copy_client.cc:647] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:37.939182 5994 tablet_copy_client.cc:523] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:37.947355 5994 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:38.011385 31199 consensus_queue.cc:237] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:38.014215 30940 raft_consensus.cc:1261] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:38.014757 31139 consensus_peers.cc:485] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a82a00c33e1412f93ab071d5b0d824a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:38.014757 30808 raft_consensus.cc:1261] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:38.015331 5763 consensus_queue.cc:1035] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:38.015873 5994 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.016270 5994 tablet_bootstrap.cc:492] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:38.016520 5994 ts_tablet_manager.cc:1367] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.069s user 0.054s sys 0.004s
I20240503 14:14:38.016446 5608 consensus_queue.cc:1035] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:38.017211 5994 raft_consensus.cc:348] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:38.017406 5994 raft_consensus.cc:729] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:38.017612 5994 consensus_queue.cc:260] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:38.018266 5994 ts_tablet_manager.cc:1398] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:38.019521 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-5f25a3bcdef849499e4a02f01f4dbc0e received from {username='slave'} at 127.20.240.70:48859
I20240503 14:14:38.019627 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-5f25a3bcdef849499e4a02f01f4dbc0e on tablet 5f25a3bcdef849499e4a02f01f4dbc0e with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:38.027840 5763 raft_consensus.cc:2932] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:38.030880 30808 raft_consensus.cc:2932] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:38.036379 30940 raft_consensus.cc:2932] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:38.039728 5996 ts_tablet_manager.cc:902] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:14:38.039820 27896 catalog_manager.cc:5484] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:38.066478 5996 tablet_copy_client.cc:308] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:14:38.067122 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet c761d663c0bd48529edc9e454d5292ac from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:42647)
I20240503 14:14:38.067281 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet c761d663c0bd48529edc9e454d5292ac from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:42647: session id = 530893a961af4b95b6b687fa5ed22f9d-c761d663c0bd48529edc9e454d5292ac
I20240503 14:14:38.069586 31092 tablet_copy_source_session.cc:205] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:38.070991 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c761d663c0bd48529edc9e454d5292ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:38.076102 5996 tablet_copy_client.cc:783] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:38.076356 5996 tablet_copy_client.cc:647] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:38.078152 5996 tablet_copy_client.cc:523] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.085100 5996 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:14:38.098119 5996 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.098457 5996 tablet_bootstrap.cc:492] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:14:38.098659 5996 ts_tablet_manager.cc:1367] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20240503 14:14:38.099244 5996 raft_consensus.cc:348] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:38.099436 5996 raft_consensus.cc:729] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:14:38.099656 5996 consensus_queue.cc:260] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:38.100231 5996 ts_tablet_manager.cc:1398] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:14:38.101064 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-c761d663c0bd48529edc9e454d5292ac received from {username='slave'} at 127.20.240.72:42647
I20240503 14:14:38.101154 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-c761d663c0bd48529edc9e454d5292ac on tablet c761d663c0bd48529edc9e454d5292ac with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:14:38.123855 30940 consensus_queue.cc:237] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:38.126195 31199 raft_consensus.cc:1261] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:38.126499 30877 consensus_peers.cc:485] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d0b457ba3564c52824f84b78cf14c23. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:38.126780 6187 consensus_queue.cc:1035] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:38.126936 31071 raft_consensus.cc:1261] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:38.127403 5295 consensus_queue.cc:1035] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:38.129675 5295 raft_consensus.cc:2932] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:38.130213 31071 raft_consensus.cc:2932] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:38.135167 31199 raft_consensus.cc:2932] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:38.135963 27896 catalog_manager.cc:5484] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:38.178864 30940 raft_consensus.cc:1203] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.186820 28303 raft_consensus.cc:1203] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:38.315809 6236 ts_tablet_manager.cc:902] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:38.344810 6236 tablet_copy_client.cc:308] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:38.345827 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet f7242c3728b84084a738fde7ae32ec0a from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:35861)
I20240503 14:14:38.345961 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet f7242c3728b84084a738fde7ae32ec0a from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:35861: session id = 5388902973134958a030fd63ab4faba4-f7242c3728b84084a738fde7ae32ec0a
I20240503 14:14:38.348385 28063 tablet_copy_source_session.cc:205] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:38.359522 6236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7242c3728b84084a738fde7ae32ec0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:38.370723 6236 tablet_copy_client.cc:783] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:38.387029 6236 tablet_copy_client.cc:647] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:38.389784 6236 tablet_copy_client.cc:523] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.401393 6236 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:38.403246 6251 ts_tablet_manager.cc:902] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:14:38.413836 30940 raft_consensus.cc:1203] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:38.418170 6236 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.420207 6236 tablet_bootstrap.cc:492] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:38.420496 6236 ts_tablet_manager.cc:1367] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20240503 14:14:38.421408 6236 raft_consensus.cc:348] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:38.421763 6236 raft_consensus.cc:729] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:38.422040 6236 consensus_queue.cc:260] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:38.422789 6236 ts_tablet_manager.cc:1398] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:38.423367 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-f7242c3728b84084a738fde7ae32ec0a received from {username='slave'} at 127.20.240.68:35861
I20240503 14:14:38.423478 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 5388902973134958a030fd63ab4faba4-f7242c3728b84084a738fde7ae32ec0a on tablet f7242c3728b84084a738fde7ae32ec0a with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:38.443431 5994 ts_tablet_manager.cc:902] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:38.459458 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:38.470801 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:38.474318 6251 tablet_copy_client.cc:308] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:14:38.476310 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet 96773dd7242d4223bdc663ee53bfdd3f from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:45597)
I20240503 14:14:38.476507 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet 96773dd7242d4223bdc663ee53bfdd3f from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:45597: session id = a4cdf05256a0407c8929ac43eb55d44f-96773dd7242d4223bdc663ee53bfdd3f
I20240503 14:14:38.487855 30830 tablet_copy_source_session.cc:205] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:38.489815 6236 ts_tablet_manager.cc:902] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:14:38.491736 6251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96773dd7242d4223bdc663ee53bfdd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:38.537833 6251 tablet_copy_client.cc:783] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:38.538129 6251 tablet_copy_client.cc:647] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:38.562142 6251 tablet_copy_client.cc:523] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.562427 6236 tablet_copy_client.cc:308] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.72:41331
I20240503 14:14:38.563004 31223 tablet_copy_service.cc:140] P 530893a961af4b95b6b687fa5ed22f9d: Received BeginTabletCopySession request for tablet 3a82a00c33e1412f93ab071d5b0d824a from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:40129)
I20240503 14:14:38.563139 31223 tablet_copy_service.cc:161] P 530893a961af4b95b6b687fa5ed22f9d: Beginning new tablet copy session on tablet 3a82a00c33e1412f93ab071d5b0d824a from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:40129: session id = 5388902973134958a030fd63ab4faba4-3a82a00c33e1412f93ab071d5b0d824a
I20240503 14:14:38.565776 31223 tablet_copy_source_session.cc:205] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:38.567201 6251 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:38.569115 6236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a82a00c33e1412f93ab071d5b0d824a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:38.580765 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:38.592463 6236 tablet_copy_client.cc:783] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:38.592694 6236 tablet_copy_client.cc:647] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:38.595234 6236 tablet_copy_client.cc:523] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.604951 6236 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:38.608671 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:38.609105 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:38.619247 6236 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.620175 6236 tablet_bootstrap.cc:492] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:38.620420 6236 ts_tablet_manager.cc:1367] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20240503 14:14:38.620558 31203 raft_consensus.cc:1203] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:38.621055 6236 raft_consensus.cc:348] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:38.622567 6236 raft_consensus.cc:729] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:38.622857 6236 consensus_queue.cc:260] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:38.623502 6236 ts_tablet_manager.cc:1398] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:14:38.624418 31223 tablet_copy_service.cc:342] P 530893a961af4b95b6b687fa5ed22f9d: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-3a82a00c33e1412f93ab071d5b0d824a received from {username='slave'} at 127.20.240.68:40129
I20240503 14:14:38.624521 31223 tablet_copy_service.cc:433] P 530893a961af4b95b6b687fa5ed22f9d: ending tablet copy session 5388902973134958a030fd63ab4faba4-3a82a00c33e1412f93ab071d5b0d824a on tablet 3a82a00c33e1412f93ab071d5b0d824a with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:38.627658 5994 tablet_copy_client.cc:308] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:38.628278 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet cb88c79b4b1a40c2b45c2dd79ab894d4 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:48859)
I20240503 14:14:38.628415 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet cb88c79b4b1a40c2b45c2dd79ab894d4 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:48859: session id = 6bbc686598f24f518d7ca6bb2063058d-cb88c79b4b1a40c2b45c2dd79ab894d4
I20240503 14:14:38.632640 28063 tablet_copy_source_session.cc:205] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:38.634071 5994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb88c79b4b1a40c2b45c2dd79ab894d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:38.650641 5994 tablet_copy_client.cc:783] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:38.650943 5994 tablet_copy_client.cc:647] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:38.657330 5994 tablet_copy_client.cc:523] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.662343 5994 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:38.703819 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:38.725865 6279 ts_tablet_manager.cc:902] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:38.728528 6251 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.728915 6251 tablet_bootstrap.cc:492] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:38.729166 6251 ts_tablet_manager.cc:1367] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.162s user 0.007s sys 0.000s
I20240503 14:14:38.729645 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:38.760386 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:38.729876 6251 raft_consensus.cc:348] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:38.768975 6251 raft_consensus.cc:729] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:38.770210 6251 consensus_queue.cc:260] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:38.774528 6251 ts_tablet_manager.cc:1398] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.045s user 0.004s sys 0.000s
I20240503 14:14:38.775111 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-96773dd7242d4223bdc663ee53bfdd3f received from {username='slave'} at 127.20.240.67:45597
I20240503 14:14:38.775205 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-96773dd7242d4223bdc663ee53bfdd3f on tablet 96773dd7242d4223bdc663ee53bfdd3f with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:38.777979 28434 raft_consensus.cc:1203] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:38.800801 5994 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.801328 5994 tablet_bootstrap.cc:492] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:38.801648 5994 ts_tablet_manager.cc:1367] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.139s user 0.052s sys 0.001s
I20240503 14:14:38.802536 5994 raft_consensus.cc:348] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:38.802845 5994 raft_consensus.cc:729] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:38.803138 5994 consensus_queue.cc:260] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:38.808763 5994 ts_tablet_manager.cc:1398] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 14:14:38.814653 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-cb88c79b4b1a40c2b45c2dd79ab894d4 received from {username='slave'} at 127.20.240.70:48859
I20240503 14:14:38.814764 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-cb88c79b4b1a40c2b45c2dd79ab894d4 on tablet cb88c79b4b1a40c2b45c2dd79ab894d4 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:38.836791 6279 tablet_copy_client.cc:308] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:38.837445 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 2d0b457ba3564c52824f84b78cf14c23 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:54091)
I20240503 14:14:38.837574 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 2d0b457ba3564c52824f84b78cf14c23 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:54091: session id = a4cdf05256a0407c8929ac43eb55d44f-2d0b457ba3564c52824f84b78cf14c23
I20240503 14:14:38.842929 30961 tablet_copy_source_session.cc:205] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:38.844252 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d0b457ba3564c52824f84b78cf14c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:38.850718 6279 tablet_copy_client.cc:783] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:38.851222 6279 tablet_copy_client.cc:647] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:38.852605 6279 tablet_copy_client.cc:523] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:38.855921 6279 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:38.861315 6279 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:38.861635 6279 tablet_bootstrap.cc:492] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:38.861850 6279 ts_tablet_manager.cc:1367] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240503 14:14:38.862504 6279 raft_consensus.cc:348] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:38.862727 6279 raft_consensus.cc:729] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:38.862952 6279 consensus_queue.cc:260] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:38.863613 6279 ts_tablet_manager.cc:1398] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:38.864470 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-2d0b457ba3564c52824f84b78cf14c23 received from {username='slave'} at 127.20.240.67:54091
I20240503 14:14:38.864568 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-2d0b457ba3564c52824f84b78cf14c23 on tablet 2d0b457ba3564c52824f84b78cf14c23 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:38.907161 28303 raft_consensus.cc:1203] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:39.002532 30940 raft_consensus.cc:1203] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:39.102088 28434 raft_consensus.cc:1203] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:39.180645 28303 raft_consensus.cc:1203] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:39.188099 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:39.204218 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:39.205495 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:39.249842 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:39.261538 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:39.264158 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:39.267755 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:39.330302 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:39.470116 31202 consensus_queue.cc:237] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:39.472276 30940 raft_consensus.cc:1261] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:39.472421 28433 raft_consensus.cc:1261] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:39.472872 5763 consensus_queue.cc:1035] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:39.473224 5763 consensus_queue.cc:1035] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:39.475508 5763 raft_consensus.cc:2932] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:39.476774 28433 raft_consensus.cc:2932] T 3a82a00c33e1412f93ab071d5b0d824a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:39.477064 30940 raft_consensus.cc:2932] T 3a82a00c33e1412f93ab071d5b0d824a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:39.482115 27895 catalog_manager.cc:5484] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:39.496153 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a82a00c33e1412f93ab071d5b0d824a with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:14:39.497002 6376 tablet_replica.cc:330] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:14:39.497313 6376 raft_consensus.cc:2229] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:39.497582 6376 raft_consensus.cc:2258] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:39.500367 6376 ts_tablet_manager.cc:1875] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:39.509479 6376 ts_tablet_manager.cc:1888] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:39.509721 6376 log.cc:1196] T 3a82a00c33e1412f93ab071d5b0d824a P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/3a82a00c33e1412f93ab071d5b0d824a
I20240503 14:14:39.510762 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 3a82a00c33e1412f93ab071d5b0d824a (table rebalancer_extra_table_38 [id=5d38544e799840a183365dc33eae8342]) successfully deleted
I20240503 14:14:39.835956 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:39.866668 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:39.876211 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:39.879632 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:39.913182 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:39.919361 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:39.947598 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:39.967207 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:40.137476 30807 consensus_queue.cc:237] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:40.139626 28433 raft_consensus.cc:1261] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:40.139971 28303 raft_consensus.cc:1261] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:40.140259 6289 consensus_queue.cc:1035] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:40.140496 4946 consensus_queue.cc:1035] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:40.143299 4946 raft_consensus.cc:2932] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:40.144078 28433 raft_consensus.cc:2932] T 96773dd7242d4223bdc663ee53bfdd3f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:40.149571 27900 catalog_manager.cc:5484] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:40.152246 28303 raft_consensus.cc:2932] T 96773dd7242d4223bdc663ee53bfdd3f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:40.166013 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 96773dd7242d4223bdc663ee53bfdd3f with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:40.177343 6461 tablet_replica.cc:330] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:40.177867 6461 raft_consensus.cc:2229] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:40.178290 6461 raft_consensus.cc:2258] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:40.178983 6461 ts_tablet_manager.cc:1875] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:40.187754 6461 ts_tablet_manager.cc:1888] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:40.188436 6461 log.cc:1196] T 96773dd7242d4223bdc663ee53bfdd3f P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/96773dd7242d4223bdc663ee53bfdd3f
I20240503 14:14:40.189152 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 96773dd7242d4223bdc663ee53bfdd3f (table rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]) successfully deleted
I20240503 14:14:40.466100 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:40.466519 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:40.487237 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:40.543501 28021 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:40.562794 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:40.605083 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:40.613107 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:40.618228 31050 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:14:40.784339 28040 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a"
dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58826
I20240503 14:14:40.784504 28040 raft_consensus.cc:595] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:40.876722 6028 raft_consensus.cc:979] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: : Instructing follower fde134d0fa2b4900b9badd2e08fea09d to start an election
I20240503 14:14:40.876845 6028 raft_consensus.cc:1067] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Signalling peer fde134d0fa2b4900b9badd2e08fea09d to start an election
I20240503 14:14:40.877434 31071 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a"
dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
from {username='slave'} at 127.20.240.65:43505
I20240503 14:14:40.877604 31071 raft_consensus.cc:482] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:40.877708 31071 raft_consensus.cc:3037] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.880406 31071 raft_consensus.cc:504] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:40.881385 31071 leader_election.cc:288] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:40.881603 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:14:40.881805 28040 raft_consensus.cc:3032] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:40.881871 28040 raft_consensus.cc:729] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:14:40.882071 31202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:14:40.882275 31202 raft_consensus.cc:3037] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.882071 28040 consensus_queue.cc:260] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:40.883212 28040 raft_consensus.cc:3037] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.884310 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7242c3728b84084a738fde7ae32ec0a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:14:40.884619 28433 raft_consensus.cc:3037] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:40.888075 28040 raft_consensus.cc:2454] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:40.891871 31202 raft_consensus.cc:2454] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:40.892383 31006 leader_election.cc:302] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:14:40.892663 6517 raft_consensus.cc:2781] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:40.892808 6517 raft_consensus.cc:686] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:14:40.893085 6517 consensus_queue.cc:237] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:40.895995 28433 raft_consensus.cc:2454] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:40.900408 27894 catalog_manager.cc:5484] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 1 to 2, leader changed from 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:41.092545 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:41.133157 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:41.133687 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:41.149654 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:41.164641 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:41.205161 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:41.224797 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:41.292827 28434 raft_consensus.cc:1261] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:41.293467 6412 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:41.306574 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:41.401861 31202 raft_consensus.cc:1261] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:41.402460 6412 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:41.428085 28040 raft_consensus.cc:1261] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:41.428635 6517 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:41.733625 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:41.759318 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:41.771512 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:41.806120 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:41.818166 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:41.833673 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:41.835852 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:41.868244 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:42.023224 31069 consensus_queue.cc:237] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:42.024976 31199 raft_consensus.cc:1261] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:42.025085 28173 raft_consensus.cc:1261] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:42.025470 6412 consensus_queue.cc:1035] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:42.025713 6513 consensus_queue.cc:1035] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:42.028486 6696 raft_consensus.cc:2932] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:42.031354 28173 raft_consensus.cc:2932] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:42.037132 27899 catalog_manager.cc:5484] T c761d663c0bd48529edc9e454d5292ac P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:42.046054 31199 raft_consensus.cc:2932] T c761d663c0bd48529edc9e454d5292ac P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:42.061111 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet c761d663c0bd48529edc9e454d5292ac with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:14:42.061651 6707 tablet_replica.cc:330] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:14:42.061833 6707 raft_consensus.cc:2229] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.062005 6707 raft_consensus.cc:2258] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.062525 6707 ts_tablet_manager.cc:1875] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.071233 6707 ts_tablet_manager.cc:1888] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:42.071357 6707 log.cc:1196] T c761d663c0bd48529edc9e454d5292ac P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/c761d663c0bd48529edc9e454d5292ac
I20240503 14:14:42.072149 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet c761d663c0bd48529edc9e454d5292ac (table rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]) successfully deleted
I20240503 14:14:42.342814 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:42.385985 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:42.389698 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:42.401352 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:42.402699 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:42.417196 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:42.418092 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:42.454780 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:42.685310 28040 consensus_queue.cc:237] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:42.687501 30940 raft_consensus.cc:1261] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.687625 28434 raft_consensus.cc:1261] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:42.688078 6100 consensus_queue.cc:1035] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.688362 6028 consensus_queue.cc:1035] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:42.690920 6100 raft_consensus.cc:2932] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:42.691403 28434 raft_consensus.cc:2932] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:42.697222 27901 catalog_manager.cc:5484] T 5f25a3bcdef849499e4a02f01f4dbc0e P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:42.710989 30940 raft_consensus.cc:2932] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:42.722328 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f25a3bcdef849499e4a02f01f4dbc0e with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:42.722916 6793 tablet_replica.cc:330] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:42.723187 6793 raft_consensus.cc:2229] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:42.724166 6793 raft_consensus.cc:2258] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:42.724790 6793 ts_tablet_manager.cc:1875] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:42.729676 6793 ts_tablet_manager.cc:1888] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:42.729835 6793 log.cc:1196] T 5f25a3bcdef849499e4a02f01f4dbc0e P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/5f25a3bcdef849499e4a02f01f4dbc0e
I20240503 14:14:42.730665 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 5f25a3bcdef849499e4a02f01f4dbc0e (table rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]) successfully deleted
I20240503 14:14:42.988682 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:43.008705 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:43.023985 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:43.040189 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:43.074590 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:43.141327 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:43.160430 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:43.232306 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:43.343218 31199 consensus_queue.cc:237] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:43.351389 28303 raft_consensus.cc:1261] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:43.351889 6607 consensus_queue.cc:1035] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:43.352871 28040 raft_consensus.cc:1261] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:43.353636 6741 consensus_queue.cc:1035] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:43.356158 6607 raft_consensus.cc:2932] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:43.356925 28303 raft_consensus.cc:2932] T e389fcad67cc4add9db2f34eb57745c8 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:43.366140 27895 catalog_manager.cc:5484] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:43.367686 28043 raft_consensus.cc:2932] T e389fcad67cc4add9db2f34eb57745c8 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:43.424012 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet e389fcad67cc4add9db2f34eb57745c8 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:14:43.436468 6882 tablet_replica.cc:330] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:14:43.436966 6882 raft_consensus.cc:2229] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:43.437273 6882 raft_consensus.cc:2258] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:43.438156 6882 ts_tablet_manager.cc:1875] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:43.448647 6882 ts_tablet_manager.cc:1888] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:43.448835 6882 log.cc:1196] T e389fcad67cc4add9db2f34eb57745c8 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/e389fcad67cc4add9db2f34eb57745c8
I20240503 14:14:43.450153 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet e389fcad67cc4add9db2f34eb57745c8 (table rebalancer_extra_table_42 [id=f15775d8335f4f6582fafa680579249f]) successfully deleted
I20240503 14:14:43.803797 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:43.813527 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:43.836266 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:43.842497 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:43.852304 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:43.860731 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:43.878612 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:43.933915 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:44.081640 28303 consensus_queue.cc:237] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:44.083603 28171 raft_consensus.cc:1261] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.084087 30939 raft_consensus.cc:1261] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:44.084124 6881 consensus_queue.cc:1035] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:44.084503 6421 consensus_queue.cc:1035] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:44.087265 6421 raft_consensus.cc:2932] T f133ebe695a74f86be1a37ae661719ec P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:44.087685 28171 raft_consensus.cc:2932] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:44.091459 30939 raft_consensus.cc:2932] T f133ebe695a74f86be1a37ae661719ec P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:44.096871 27894 catalog_manager.cc:5484] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:44.120748 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet f133ebe695a74f86be1a37ae661719ec with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52616
I20240503 14:14:44.123874 6977 tablet_replica.cc:330] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:14:44.124130 6977 raft_consensus.cc:2229] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:44.124329 6977 raft_consensus.cc:2258] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:44.124953 6977 ts_tablet_manager.cc:1875] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:44.129912 6977 ts_tablet_manager.cc:1888] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:44.130016 6977 log.cc:1196] T f133ebe695a74f86be1a37ae661719ec P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/f133ebe695a74f86be1a37ae661719ec
I20240503 14:14:44.131600 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet f133ebe695a74f86be1a37ae661719ec (table rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]) successfully deleted
I20240503 14:14:44.489751 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:44.507668 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:44.512166 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:44.560178 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:44.644409 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:44.666020 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:44.669087 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:44.742169 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:44.830199 28303 consensus_queue.cc:237] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:44.832067 28433 raft_consensus.cc:1261] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:44.832145 28043 raft_consensus.cc:1261] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:44.832527 6608 consensus_queue.cc:1035] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:44.832834 6881 consensus_queue.cc:1035] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:44.835979 6608 raft_consensus.cc:2932] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:44.837659 28043 raft_consensus.cc:2932] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:44.845635 27902 catalog_manager.cc:5484] T 8baf804d83954d288576c313efe3d1cb P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:44.850889 28433 raft_consensus.cc:2932] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:44.860965 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 8baf804d83954d288576c313efe3d1cb with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:44.862388 7062 tablet_replica.cc:330] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:44.862601 7062 raft_consensus.cc:2229] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:44.862993 7062 raft_consensus.cc:2258] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:44.863657 7062 ts_tablet_manager.cc:1875] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:44.872960 7062 ts_tablet_manager.cc:1888] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:44.873054 7062 log.cc:1196] T 8baf804d83954d288576c313efe3d1cb P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/8baf804d83954d288576c313efe3d1cb
I20240503 14:14:44.873773 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 8baf804d83954d288576c313efe3d1cb (table rebalancer_extra_table_10 [id=b41da97c57504b3ab03a74308af97dee]) successfully deleted
I20240503 14:14:45.133684 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:45.134810 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:45.172736 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:45.179563 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:45.189054 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:45.215329 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:45.216274 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:45.234385 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:14:45.277638 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 116821us
W20240503 14:14:45.305294 31008 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.67:43889, user_credentials={real_user=slave}} blocked reactor thread for 82645.9us
I20240503 14:14:45.465556 28173 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a"
dest_uuid: "6409961ee8c0445687c11724213c44a9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36054
I20240503 14:14:45.465785 28173 raft_consensus.cc:595] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:45.699059 6190 raft_consensus.cc:979] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: : Instructing follower fde134d0fa2b4900b9badd2e08fea09d to start an election
I20240503 14:14:45.699204 6190 raft_consensus.cc:1067] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Signalling peer fde134d0fa2b4900b9badd2e08fea09d to start an election
I20240503 14:14:45.700167 31072 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a"
dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
from {username='slave'} at 127.20.240.66:42465
I20240503 14:14:45.700358 31072 raft_consensus.cc:482] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:45.700446 31072 raft_consensus.cc:3037] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:45.703369 31072 raft_consensus.cc:504] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:45.712011 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:45.713730 30939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:14:45.713919 30939 raft_consensus.cc:3037] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:45.716678 30939 raft_consensus.cc:2454] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:45.735905 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:14:45.736128 28173 raft_consensus.cc:3032] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:45.736200 28173 raft_consensus.cc:729] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:14:45.736526 28173 consensus_queue.cc:260] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:45.736984 28173 raft_consensus.cc:3037] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:45.742766 28173 raft_consensus.cc:2454] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:45.743292 31006 leader_election.cc:302] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:14:45.743842 7128 raft_consensus.cc:2781] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:45.744005 7128 raft_consensus.cc:686] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:14:45.744359 7128 consensus_queue.cc:237] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:45.749588 31072 leader_election.cc:288] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:14:45.749969 30807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abfcd38ecc64acfac35c676b941581a" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:14:45.750169 30807 raft_consensus.cc:3037] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:45.753203 30807 raft_consensus.cc:2454] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:14:45.788096 27901 catalog_manager.cc:5484] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 1 to 2, leader changed from 6409961ee8c0445687c11724213c44a9 (127.20.240.66) to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:45.833433 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:45.880674 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:45.897964 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:45.903272 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:45.930303 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:45.989148 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:46.017467 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:46.150645 28172 raft_consensus.cc:1261] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:46.151213 7128 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:46.167315 30940 raft_consensus.cc:1261] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:46.167806 7128 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:46.180322 30807 raft_consensus.cc:1261] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:46.183131 7126 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:46.386494 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:46.408419 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:46.415273 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:46.418766 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:46.447961 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:46.453104 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:46.507457 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:46.561139 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:46.707501 28432 consensus_queue.cc:237] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:46.709223 28172 raft_consensus.cc:1261] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:46.709244 31069 raft_consensus.cc:1261] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:46.709681 7204 consensus_queue.cc:1035] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:46.709959 6834 consensus_queue.cc:1035] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:46.712275 6834 raft_consensus.cc:2932] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:46.712813 28172 raft_consensus.cc:2932] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:46.713192 31069 raft_consensus.cc:2932] T ac08c6dba4dc4e90ab11a155ec6ecdfd P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:46.718750 27902 catalog_manager.cc:5484] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:46.734987 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet ac08c6dba4dc4e90ab11a155ec6ecdfd with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43626
I20240503 14:14:46.735826 7298 tablet_replica.cc:330] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:14:46.736119 7298 raft_consensus.cc:2229] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:46.736447 7298 raft_consensus.cc:2258] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:46.737279 7298 ts_tablet_manager.cc:1875] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:46.743389 7298 ts_tablet_manager.cc:1888] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:46.743541 7298 log.cc:1196] T ac08c6dba4dc4e90ab11a155ec6ecdfd P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/ac08c6dba4dc4e90ab11a155ec6ecdfd
I20240503 14:14:46.744256 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet ac08c6dba4dc4e90ab11a155ec6ecdfd (table rebalance_test_table_1 [id=ca9ae4dcdb734b4caba55748be1857a2]) successfully deleted
I20240503 14:14:47.053241 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:47.068336 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:47.073750 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:47.076864 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:47.084748 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:47.251092 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:47.252420 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:47.255007 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:47.432955 28043 consensus_queue.cc:237] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:47.434576 30941 raft_consensus.cc:1261] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:47.434648 28434 raft_consensus.cc:1261] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:47.435082 7195 consensus_queue.cc:1035] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:47.435508 6100 consensus_queue.cc:1035] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:47.438375 6100 raft_consensus.cc:2932] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:47.442880 30940 raft_consensus.cc:2932] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:47.449690 27895 catalog_manager.cc:5484] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:47.462464 28434 raft_consensus.cc:2932] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:47.473707 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet cb88c79b4b1a40c2b45c2dd79ab894d4 with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:47.474540 7396 tablet_replica.cc:330] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:47.474828 7396 raft_consensus.cc:2229] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:47.475217 7396 raft_consensus.cc:2258] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:47.478394 7396 ts_tablet_manager.cc:1875] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:47.484191 7396 ts_tablet_manager.cc:1888] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:47.484294 7396 log.cc:1196] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/cb88c79b4b1a40c2b45c2dd79ab894d4
I20240503 14:14:47.484881 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet cb88c79b4b1a40c2b45c2dd79ab894d4 (table rebalance_test_table_17 [id=4a94574e1a5b44909f1e81be5a6767f9]) successfully deleted
I20240503 14:14:47.781177 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:47.787449 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:47.812928 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:47.817057 28154 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:47.820608 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:47.867837 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:47.930977 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:47.934525 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:48.031999 28041 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562"
dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58992
I20240503 14:14:48.032181 28041 raft_consensus.cc:595] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:14:48.248191 7195 raft_consensus.cc:979] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: : Instructing follower 6409961ee8c0445687c11724213c44a9 to start an election
I20240503 14:14:48.248342 7195 raft_consensus.cc:1067] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Signalling peer 6409961ee8c0445687c11724213c44a9 to start an election
I20240503 14:14:48.251554 28172 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562"
dest_uuid: "6409961ee8c0445687c11724213c44a9"
from {username='slave'} at 127.20.240.65:50673
I20240503 14:14:48.251731 28172 raft_consensus.cc:482] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:14:48.251817 28172 raft_consensus.cc:3037] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.257382 28172 raft_consensus.cc:504] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:48.258430 28172 leader_election.cc:288] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:48.258396 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:14:48.258581 28041 raft_consensus.cc:3032] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:14:48.258671 28041 raft_consensus.cc:729] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:14:48.258618 28303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:14:48.258792 28303 raft_consensus.cc:3037] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.258916 28041 consensus_queue.cc:260] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:48.259321 28041 raft_consensus.cc:3037] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.261328 28303 raft_consensus.cc:2454] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:14:48.261961 28041 raft_consensus.cc:2454] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:14:48.262369 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38cf3aab9e1b4f28842ac16d02be2562" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:14:48.262603 30941 raft_consensus.cc:3037] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:14:48.262897 28110 leader_election.cc:302] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:14:48.263131 7365 raft_consensus.cc:2781] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:14:48.263254 7365 raft_consensus.cc:686] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:14:48.263495 7365 consensus_queue.cc:237] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:48.265218 30941 raft_consensus.cc:2454] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:14:48.286002 27901 catalog_manager.cc:5484] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 1 to 2, leader changed from 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:48.351825 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:48.372355 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:48.375661 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:48.418748 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:48.425235 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:48.547068 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:48.557626 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:48.558805 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:48.814878 28041 raft_consensus.cc:1261] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:48.816437 7453 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:48.836037 28302 raft_consensus.cc:1261] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:48.836477 7365 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:14:48.845633 30877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.65:33293, user_credentials={real_user=slave}} blocked reactor thread for 48889.3us
I20240503 14:14:48.847824 30941 raft_consensus.cc:1261] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:14:48.848925 7365 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:49.052423 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:49.109998 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:49.126505 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:14:49.135952 31137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 75645us
I20240503 14:14:49.142455 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:49.151461 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:49.181021 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:49.193285 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:49.195243 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:49.411612 30941 consensus_queue.cc:237] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:49.413291 28043 raft_consensus.cc:1261] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:49.413317 28433 raft_consensus.cc:1261] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:49.413748 7364 consensus_queue.cc:1035] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:49.414211 7364 consensus_queue.cc:1035] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:49.416227 7364 raft_consensus.cc:2932] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:49.419248 28433 raft_consensus.cc:2932] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:49.421393 28043 raft_consensus.cc:2932] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:49.422611 27899 catalog_manager.cc:5484] T 1f4fbfaadc474d59bfb23107f0342f7c P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:49.437359 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4fbfaadc474d59bfb23107f0342f7c with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:14:49.438807 7632 tablet_replica.cc:330] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:14:49.439188 7632 raft_consensus.cc:2229] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:49.439512 7632 raft_consensus.cc:2258] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:49.440354 7632 ts_tablet_manager.cc:1875] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:49.448124 7632 ts_tablet_manager.cc:1888] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:49.448388 7632 log.cc:1196] T 1f4fbfaadc474d59bfb23107f0342f7c P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/1f4fbfaadc474d59bfb23107f0342f7c
I20240503 14:14:49.449344 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 1f4fbfaadc474d59bfb23107f0342f7c (table rebalancer_extra_table_58 [id=1d8deea75eed40969c9d8a4b510bbe80]) successfully deleted
I20240503 14:14:49.743335 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:49.749796 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:49.762758 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:49.777741 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:49.800674 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:49.808022 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:49.820199 27891 maintenance_manager.cc:419] P 5baa17487b4b4e1eb60d2f5fcb4de355: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033378
I20240503 14:14:49.835857 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:49.871754 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:50.011770 27890 maintenance_manager.cc:643] P 5baa17487b4b4e1eb60d2f5fcb4de355: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.191s user 0.086s sys 0.003s Metrics: {"bytes_written":105201,"cfile_init":1,"dirs.queue_time_us":13725,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":4195,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1550,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":291,"thread_start_us":8040,"threads_started":1}
I20240503 14:14:50.013096 27891 maintenance_manager.cc:419] P 5baa17487b4b4e1eb60d2f5fcb4de355: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 32658 bytes on disk
I20240503 14:14:50.013801 27890 maintenance_manager.cc:643] P 5baa17487b4b4e1eb60d2f5fcb4de355: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 14:14:50.035987 28433 consensus_queue.cc:237] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:50.037815 28302 raft_consensus.cc:1261] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:50.038369 7343 consensus_queue.cc:1035] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:50.038481 31199 raft_consensus.cc:1261] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:50.039026 7343 consensus_queue.cc:1035] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:50.040967 7343 raft_consensus.cc:2932] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:50.041296 28302 raft_consensus.cc:2932] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:50.047214 27893 catalog_manager.cc:5484] T 664cadf5232f4768adf6438e57f7b7c9 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:50.065632 31199 raft_consensus.cc:2932] T 664cadf5232f4768adf6438e57f7b7c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:50.095908 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 664cadf5232f4768adf6438e57f7b7c9 with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:50.118618 7717 tablet_replica.cc:330] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:50.118911 7717 raft_consensus.cc:2229] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:50.119135 7717 raft_consensus.cc:2258] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:50.119704 7717 ts_tablet_manager.cc:1875] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:50.130051 7717 ts_tablet_manager.cc:1888] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:50.130187 7717 log.cc:1196] T 664cadf5232f4768adf6438e57f7b7c9 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/664cadf5232f4768adf6438e57f7b7c9
I20240503 14:14:50.131264 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 664cadf5232f4768adf6438e57f7b7c9 (table rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]) successfully deleted
I20240503 14:14:50.492933 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:50.536387 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:50.569849 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:50.576237 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:50.633819 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:50.652837 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:50.654683 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:50.710829 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:50.815829 28171 consensus_queue.cc:237] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:50.817817 28041 raft_consensus.cc:1261] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:50.817834 31072 raft_consensus.cc:1261] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:50.818365 7762 consensus_queue.cc:1035] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:50.818672 7761 consensus_queue.cc:1035] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:50.820814 7453 raft_consensus.cc:2932] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:50.821328 28041 raft_consensus.cc:2932] T a816d34a63b945b08eec771d9e7c2d55 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:50.822460 31072 raft_consensus.cc:2932] T a816d34a63b945b08eec771d9e7c2d55 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:50.826521 27893 catalog_manager.cc:5484] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:50.845973 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet a816d34a63b945b08eec771d9e7c2d55 with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43626
I20240503 14:14:50.846536 7808 tablet_replica.cc:330] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:14:50.846884 7808 raft_consensus.cc:2229] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:50.847173 7808 raft_consensus.cc:2258] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:50.847957 7808 ts_tablet_manager.cc:1875] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:50.853590 7808 ts_tablet_manager.cc:1888] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:50.853718 7808 log.cc:1196] T a816d34a63b945b08eec771d9e7c2d55 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/a816d34a63b945b08eec771d9e7c2d55
I20240503 14:14:50.854403 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet a816d34a63b945b08eec771d9e7c2d55 (table rebalance_test_table_5 [id=9c30e6ab7d16493493d8c70e665de3fd]) successfully deleted
I20240503 14:14:51.119263 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:51.131811 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:51.186911 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:51.195806 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:51.206652 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:51.232475 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:51.294554 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:51.322027 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:51.434923 28171 consensus_queue.cc:237] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:51.436673 30810 raft_consensus.cc:1261] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:51.436769 28040 raft_consensus.cc:1261] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:51.437224 7453 consensus_queue.cc:1035] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:51.437530 7762 consensus_queue.cc:1035] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:51.440383 7453 raft_consensus.cc:2932] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:51.443986 30810 raft_consensus.cc:2932] T d0d2600fd84a4d839dc3366b08485e42 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:51.445314 28040 raft_consensus.cc:2932] T d0d2600fd84a4d839dc3366b08485e42 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:51.446676 27893 catalog_manager.cc:5484] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:51.452564 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet d0d2600fd84a4d839dc3366b08485e42 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:14:51.465191 7887 tablet_replica.cc:330] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:14:51.465376 7887 raft_consensus.cc:2229] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:51.465554 7887 raft_consensus.cc:2258] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:51.466073 7887 ts_tablet_manager.cc:1875] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:51.471035 7887 ts_tablet_manager.cc:1888] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:51.471168 7887 log.cc:1196] T d0d2600fd84a4d839dc3366b08485e42 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/d0d2600fd84a4d839dc3366b08485e42
I20240503 14:14:51.471931 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet d0d2600fd84a4d839dc3366b08485e42 (table rebalancer_extra_table_46 [id=a2a1f19d4ee84199b4c413c858081638]) successfully deleted
I20240503 14:14:51.545512 28433 consensus_queue.cc:237] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:51.548481 31202 raft_consensus.cc:1261] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:51.548636 28040 raft_consensus.cc:1261] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:51.548677 28368 consensus_peers.cc:485] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 710e09d5ab4e457fbbf54b3acaf5ff15. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:51.549065 7344 consensus_queue.cc:1035] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:51.549309 7343 consensus_queue.cc:1035] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:51.557973 7344 raft_consensus.cc:2932] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:51.564145 31202 raft_consensus.cc:2932] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:51.565819 27901 catalog_manager.cc:5484] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:51.567101 28040 raft_consensus.cc:2932] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:51.600296 28171 consensus_queue.cc:237] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:51.602841 30809 raft_consensus.cc:1261] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:51.603353 7881 consensus_queue.cc:1035] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:51.603663 31202 raft_consensus.cc:1261] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:51.604125 28110 consensus_peers.cc:485] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98d39010d3e24d9fbd33c559568f7330. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:51.604434 7881 consensus_queue.cc:1035] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:51.606554 7882 raft_consensus.cc:2932] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:51.609768 30809 raft_consensus.cc:2932] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:51.609791 31202 raft_consensus.cc:2932] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:51.612318 27901 catalog_manager.cc:5484] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:52.012768 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:52.022418 7999 ts_tablet_manager.cc:902] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:52.034817 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:52.077122 8002 ts_tablet_manager.cc:902] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:52.108316 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:52.121738 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:52.197366 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:52.219450 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:52.225589 8002 tablet_copy_client.cc:308] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:52.226759 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:52.227212 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:52.227916 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 710e09d5ab4e457fbbf54b3acaf5ff15 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:55357)
I20240503 14:14:52.228055 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 710e09d5ab4e457fbbf54b3acaf5ff15 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:55357: session id = 6409961ee8c0445687c11724213c44a9-710e09d5ab4e457fbbf54b3acaf5ff15
I20240503 14:14:52.234428 28454 tablet_copy_source_session.cc:205] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:52.235951 8002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 710e09d5ab4e457fbbf54b3acaf5ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:52.277650 7999 tablet_copy_client.cc:308] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:52.278671 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet 98d39010d3e24d9fbd33c559568f7330 from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:42383)
I20240503 14:14:52.278812 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet 98d39010d3e24d9fbd33c559568f7330 from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:42383: session id = 5388902973134958a030fd63ab4faba4-98d39010d3e24d9fbd33c559568f7330
I20240503 14:14:52.281504 28194 tablet_copy_source_session.cc:205] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:52.282757 7999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d39010d3e24d9fbd33c559568f7330. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:52.290763 8002 tablet_copy_client.cc:783] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:52.295759 8002 tablet_copy_client.cc:647] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:52.296523 7999 tablet_copy_client.cc:783] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:52.296787 7999 tablet_copy_client.cc:647] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:52.298904 8002 tablet_copy_client.cc:523] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:52.300993 7999 tablet_copy_client.cc:523] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:52.307034 8002 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:14:52.314775 7999 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:14:52.328310 8002 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:52.328714 8002 tablet_bootstrap.cc:492] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:14:52.329042 8002 ts_tablet_manager.cc:1367] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.008s
I20240503 14:14:52.329777 8002 raft_consensus.cc:348] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:52.330224 8002 raft_consensus.cc:729] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:14:52.330533 8002 consensus_queue.cc:260] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:52.334436 8002 ts_tablet_manager.cc:1398] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240503 14:14:52.335062 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-710e09d5ab4e457fbbf54b3acaf5ff15 received from {username='slave'} at 127.20.240.66:55357
I20240503 14:14:52.335179 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session 6409961ee8c0445687c11724213c44a9-710e09d5ab4e457fbbf54b3acaf5ff15 on tablet 710e09d5ab4e457fbbf54b3acaf5ff15 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:14:52.338541 7999 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:52.338907 7999 tablet_bootstrap.cc:492] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:14:52.339129 7999 ts_tablet_manager.cc:1367] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.002s
I20240503 14:14:52.339789 7999 raft_consensus.cc:348] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:52.339987 7999 raft_consensus.cc:729] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:14:52.340516 7999 consensus_queue.cc:260] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:52.341154 7999 ts_tablet_manager.cc:1398] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:52.341840 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-98d39010d3e24d9fbd33c559568f7330 received from {username='slave'} at 127.20.240.68:42383
I20240503 14:14:52.341943 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 5388902973134958a030fd63ab4faba4-98d39010d3e24d9fbd33c559568f7330 on tablet 98d39010d3e24d9fbd33c559568f7330 with peer 5388902973134958a030fd63ab4faba4
I20240503 14:14:52.439389 30941 consensus_queue.cc:237] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:52.443843 28302 raft_consensus.cc:1261] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:52.443907 31072 raft_consensus.cc:1261] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:52.444396 7763 consensus_queue.cc:1035] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:52.444644 8004 consensus_queue.cc:1035] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:52.447167 7763 raft_consensus.cc:2932] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:52.447594 31072 raft_consensus.cc:2932] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:52.453243 27893 catalog_manager.cc:5484] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:52.464443 28302 raft_consensus.cc:2932] T 2d0b457ba3564c52824f84b78cf14c23 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:52.485167 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d0b457ba3564c52824f84b78cf14c23 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:14:52.495258 8049 tablet_replica.cc:330] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:14:52.495618 8049 raft_consensus.cc:2229] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:52.495925 8049 raft_consensus.cc:2258] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:52.496609 8049 ts_tablet_manager.cc:1875] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:52.503540 8049 ts_tablet_manager.cc:1888] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:52.503726 8049 log.cc:1196] T 2d0b457ba3564c52824f84b78cf14c23 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/2d0b457ba3564c52824f84b78cf14c23
I20240503 14:14:52.504251 28171 raft_consensus.cc:1203] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:52.505241 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 2d0b457ba3564c52824f84b78cf14c23 (table rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]) successfully deleted
I20240503 14:14:52.565474 28434 raft_consensus.cc:1203] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:52.818336 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:52.838214 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:52.876492 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:52.894188 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:52.909086 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:52.988835 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:53.018496 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:53.070724 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:53.164417 28433 consensus_queue.cc:237] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:14:53.166595 31202 raft_consensus.cc:1261] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:53.166754 28171 raft_consensus.cc:1261] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:53.167846 7344 consensus_queue.cc:1035] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:53.168347 7344 consensus_queue.cc:1035] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:53.180557 7344 raft_consensus.cc:2932] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:53.181171 31202 raft_consensus.cc:2932] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:53.182415 28171 raft_consensus.cc:2932] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:53.187968 27895 catalog_manager.cc:5484] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:14:53.196035 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 710e09d5ab4e457fbbf54b3acaf5ff15 with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52616
I20240503 14:14:53.196549 8137 tablet_replica.cc:330] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:14:53.196730 8137 raft_consensus.cc:2229] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:53.196895 8137 raft_consensus.cc:2258] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:53.197536 8137 ts_tablet_manager.cc:1875] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:53.203162 8137 ts_tablet_manager.cc:1888] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:53.203274 8137 log.cc:1196] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/710e09d5ab4e457fbbf54b3acaf5ff15
I20240503 14:14:53.204089 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 710e09d5ab4e457fbbf54b3acaf5ff15 (table rebalancer_extra_table_16 [id=9a283744e39a4b71b679788b2f5d2aca]) successfully deleted
I20240503 14:14:53.557550 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:53.558962 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:53.588346 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:53.617946 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:53.667152 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:53.688212 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:53.699524 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:53.729221 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:53.864675 28171 consensus_queue.cc:237] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:53.866374 28433 raft_consensus.cc:1261] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:53.866415 31199 raft_consensus.cc:1261] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:14:53.866891 7881 consensus_queue.cc:1035] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:53.867168 7762 consensus_queue.cc:1035] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:14:53.870146 7881 raft_consensus.cc:2932] T 98d39010d3e24d9fbd33c559568f7330 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:53.870609 28433 raft_consensus.cc:2932] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:53.874393 31199 raft_consensus.cc:2932] T 98d39010d3e24d9fbd33c559568f7330 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:53.877753 27898 catalog_manager.cc:5484] T 98d39010d3e24d9fbd33c559568f7330 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:53.919934 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 98d39010d3e24d9fbd33c559568f7330 with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:14:53.931002 8242 tablet_replica.cc:330] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:14:53.931216 8242 raft_consensus.cc:2229] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:53.931401 8242 raft_consensus.cc:2258] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:53.931967 8242 ts_tablet_manager.cc:1875] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:53.947335 8242 ts_tablet_manager.cc:1888] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:14:53.948931 8242 log.cc:1196] T 98d39010d3e24d9fbd33c559568f7330 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/98d39010d3e24d9fbd33c559568f7330
I20240503 14:14:53.950300 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 98d39010d3e24d9fbd33c559568f7330 (table rebalancer_extra_table_34 [id=06e77778ef6142a5bfa4db225f6cb559]) successfully deleted
I20240503 14:14:54.196413 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:54.204437 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:54.247763 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:54.262199 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:54.271296 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:54.281553 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:54.285045 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:54.414461 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:54.525422 31069 consensus_queue.cc:237] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:14:54.527357 31202 raft_consensus.cc:1261] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:54.527509 28433 raft_consensus.cc:1261] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:54.527954 7764 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:54.528308 7600 consensus_queue.cc:1035] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:54.532138 7764 raft_consensus.cc:2932] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:54.536976 28433 raft_consensus.cc:2932] T f7242c3728b84084a738fde7ae32ec0a P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:54.548691 27900 catalog_manager.cc:5484] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:54.564811 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet f7242c3728b84084a738fde7ae32ec0a with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52616
I20240503 14:14:54.568547 31202 raft_consensus.cc:2932] T f7242c3728b84084a738fde7ae32ec0a P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:14:54.576709 8324 tablet_replica.cc:330] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:14:54.576866 8324 raft_consensus.cc:2229] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:54.576978 8324 raft_consensus.cc:2258] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:54.577615 8324 ts_tablet_manager.cc:1875] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:54.584712 8324 ts_tablet_manager.cc:1888] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:14:54.584916 8324 log.cc:1196] T f7242c3728b84084a738fde7ae32ec0a P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7242c3728b84084a738fde7ae32ec0a
I20240503 14:14:54.585991 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet f7242c3728b84084a738fde7ae32ec0a (table rebalancer_extra_table_28 [id=c6f2e55142d041c5802f6d565f7fbe39]) successfully deleted
I20240503 14:14:54.865854 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:54.891494 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:54.919919 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:54.921634 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:54.965668 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:54.978917 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:54.988929 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:54.994213 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:55.156584 31069 consensus_queue.cc:237] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:55.158641 30810 raft_consensus.cc:1261] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:55.158751 30941 raft_consensus.cc:1261] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:55.159340 7600 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:55.159723 7600 consensus_queue.cc:1035] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:55.161550 7600 raft_consensus.cc:2932] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:55.162642 30810 raft_consensus.cc:2932] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:55.165243 30941 raft_consensus.cc:2932] T 0abfcd38ecc64acfac35c676b941581a P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:55.170042 27901 catalog_manager.cc:5484] T 0abfcd38ecc64acfac35c676b941581a P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index 2 to 4, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:55.199723 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 0abfcd38ecc64acfac35c676b941581a with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35238
I20240503 14:14:55.200549 8433 tablet_replica.cc:330] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:14:55.200865 8433 raft_consensus.cc:2229] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:55.201452 8433 raft_consensus.cc:2258] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:55.202186 8433 ts_tablet_manager.cc:1875] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:55.209152 8433 ts_tablet_manager.cc:1888] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:14:55.209373 8433 log.cc:1196] T 0abfcd38ecc64acfac35c676b941581a P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/0abfcd38ecc64acfac35c676b941581a
I20240503 14:14:55.210703 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 0abfcd38ecc64acfac35c676b941581a (table rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]) successfully deleted
I20240503 14:14:55.520509 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:55.522871 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:55.523209 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:55.574280 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:55.625792 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:55.630785 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:55.645082 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:55.687530 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:55.813170 28172 consensus_queue.cc:237] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:55.815268 30940 raft_consensus.cc:1261] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:55.815599 28302 raft_consensus.cc:1261] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:14:55.815868 7762 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:55.816167 8235 consensus_queue.cc:1035] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:55.818596 8235 raft_consensus.cc:2932] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:55.820549 30940 raft_consensus.cc:2932] T 38cf3aab9e1b4f28842ac16d02be2562 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:55.823578 28302 raft_consensus.cc:2932] T 38cf3aab9e1b4f28842ac16d02be2562 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:55.825331 27900 catalog_manager.cc:5484] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:55.834381 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 38cf3aab9e1b4f28842ac16d02be2562 with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52616
I20240503 14:14:55.848840 8538 tablet_replica.cc:330] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:14:55.849095 8538 raft_consensus.cc:2229] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:55.849328 8538 raft_consensus.cc:2258] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:55.849967 8538 ts_tablet_manager.cc:1875] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:55.859369 8538 ts_tablet_manager.cc:1888] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:14:55.859544 8538 log.cc:1196] T 38cf3aab9e1b4f28842ac16d02be2562 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/38cf3aab9e1b4f28842ac16d02be2562
I20240503 14:14:55.860230 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 38cf3aab9e1b4f28842ac16d02be2562 (table rebalancer_extra_table_54 [id=bfce78d2daa54f6d9ab83a550a78fd8f]) successfully deleted
I20240503 14:14:55.969288 31202 consensus_queue.cc:237] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:55.971752 28434 raft_consensus.cc:1261] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:55.972016 30940 raft_consensus.cc:1261] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:55.972368 31138 consensus_peers.cc:485] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf316cde5631415faecbc1bb26690b79. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:55.972659 8332 consensus_queue.cc:1035] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:55.973258 8477 consensus_queue.cc:1035] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:55.983424 8332 raft_consensus.cc:2932] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:55.983867 30940 raft_consensus.cc:2932] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:55.987762 28434 raft_consensus.cc:2932] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:55.989974 27895 catalog_manager.cc:5484] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.024392 30940 consensus_queue.cc:237] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:56.026813 30809 raft_consensus.cc:1261] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:56.026834 31202 raft_consensus.cc:1261] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:56.027396 8595 consensus_queue.cc:1035] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:56.027716 8404 consensus_queue.cc:1035] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:56.027980 30877 consensus_peers.cc:485] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 619951b364d44766a2f3064222ceb738. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.031154 8595 raft_consensus.cc:2932] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.031672 30809 raft_consensus.cc:2932] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.034651 31202 raft_consensus.cc:2932] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.038790 27894 catalog_manager.cc:5484] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.070628 30940 consensus_queue.cc:237] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.073156 30809 raft_consensus.cc:1261] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:56.073568 28434 raft_consensus.cc:1261] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:56.073647 30876 consensus_peers.cc:485] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee914d743a2c46418f79728f3d32d1c5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.073761 8004 consensus_queue.cc:1035] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:56.074522 8404 consensus_queue.cc:1035] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:56.087002 8404 raft_consensus.cc:2932] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.091261 28434 raft_consensus.cc:2932] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.091841 30809 raft_consensus.cc:2932] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.098178 27895 catalog_manager.cc:5484] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 2, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.144162 28172 consensus_queue.cc:237] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:56.146755 28434 raft_consensus.cc:1261] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.146955 28040 raft_consensus.cc:1261] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:56.147217 28109 consensus_peers.cc:485] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0225203f2de2498eb448e066ce545a64. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.147255 7762 consensus_queue.cc:1035] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.147516 8235 consensus_queue.cc:1035] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.150022 7762 raft_consensus.cc:2932] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.150696 28040 raft_consensus.cc:2932] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.154137 28434 raft_consensus.cc:2932] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.156638 27895 catalog_manager.cc:5484] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.209558 28434 consensus_queue.cc:237] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.212113 28304 raft_consensus.cc:1261] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.212311 28171 raft_consensus.cc:1261] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:56.212450 28369 consensus_peers.cc:485] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 213bc70b5cf14440ab88c754df0b8f7f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.212833 8653 consensus_queue.cc:1035] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.213165 7344 consensus_queue.cc:1035] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.227380 8653 raft_consensus.cc:2932] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.227909 28304 raft_consensus.cc:2932] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.230690 28171 raft_consensus.cc:2932] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.232990 27900 catalog_manager.cc:5484] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.301131 30810 consensus_queue.cc:237] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:56.303328 31202 raft_consensus.cc:1261] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:56.303753 31070 raft_consensus.cc:1261] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:56.303953 8513 consensus_queue.cc:1035] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:56.304476 8513 consensus_queue.cc:1035] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:14:56.305075 30746 consensus_peers.cc:485] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5be4220923ed4c5388a7a15755cb685d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.307117 8513 raft_consensus.cc:2932] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.307623 31070 raft_consensus.cc:2932] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.310400 31202 raft_consensus.cc:2932] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.313730 27895 catalog_manager.cc:5484] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:14:56.363869 28434 consensus_queue.cc:237] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
W20240503 14:14:56.368072 28367 consensus_peers.cc:485] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 823dde634aac449ea8cd21d38ecfb22b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.368671 28304 raft_consensus.cc:1261] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.369154 7344 consensus_queue.cc:1035] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.369518 28040 raft_consensus.cc:1261] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.369930 7344 consensus_queue.cc:1035] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.383487 8653 raft_consensus.cc:2932] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:56.384814 28040 raft_consensus.cc:2932] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:56.390707 28304 raft_consensus.cc:2932] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:56.397074 27900 catalog_manager.cc:5484] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:14:56.463752 28040 consensus_queue.cc:237] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:56.466149 28433 raft_consensus.cc:1261] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.466321 28171 raft_consensus.cc:1261] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.466745 8106 consensus_queue.cc:1035] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:14:56.467029 27978 consensus_peers.cc:485] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc9c29880eea4cd9a11eb24eeb3bd228. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.467291 8106 consensus_queue.cc:1035] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.479357 8106 raft_consensus.cc:2932] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.479919 28171 raft_consensus.cc:2932] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.487035 27900 catalog_manager.cc:5484] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.487983 28433 raft_consensus.cc:2932] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:56.496083 8716 ts_tablet_manager.cc:902] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:56.508137 8716 tablet_copy_client.cc:308] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:56.508687 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet ee914d743a2c46418f79728f3d32d1c5 from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:42719)
I20240503 14:14:56.508832 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet ee914d743a2c46418f79728f3d32d1c5 from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:42719: session id = fde134d0fa2b4900b9badd2e08fea09d-ee914d743a2c46418f79728f3d32d1c5
I20240503 14:14:56.511523 30961 tablet_copy_source_session.cc:205] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:56.516006 8716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee914d743a2c46418f79728f3d32d1c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:56.525982 8716 tablet_copy_client.cc:783] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:56.526208 8716 tablet_copy_client.cc:647] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:56.528492 8716 tablet_copy_client.cc:523] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:56.532971 8716 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:56.538470 8727 ts_tablet_manager.cc:902] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:14:56.544517 8716 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:56.544948 8716 tablet_bootstrap.cc:492] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:56.545264 8716 ts_tablet_manager.cc:1367] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.010s
I20240503 14:14:56.545976 8716 raft_consensus.cc:348] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.546222 8716 raft_consensus.cc:729] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:56.546478 8716 consensus_queue.cc:260] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.547127 8716 ts_tablet_manager.cc:1398] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:14:56.547745 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-ee914d743a2c46418f79728f3d32d1c5 received from {username='slave'} at 127.20.240.71:42719
I20240503 14:14:56.547844 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-ee914d743a2c46418f79728f3d32d1c5 on tablet ee914d743a2c46418f79728f3d32d1c5 with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:56.552836 8732 ts_tablet_manager.cc:902] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:14:56.554468 8732 tablet_copy_client.cc:308] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:14:56.560186 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet 0225203f2de2498eb448e066ce545a64 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:46183)
I20240503 14:14:56.560335 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet 0225203f2de2498eb448e066ce545a64 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:46183: session id = 6bbc686598f24f518d7ca6bb2063058d-0225203f2de2498eb448e066ce545a64
I20240503 14:14:56.563017 28194 tablet_copy_source_session.cc:205] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:56.565510 8732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225203f2de2498eb448e066ce545a64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:56.572283 8732 tablet_copy_client.cc:783] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:56.572620 8732 tablet_copy_client.cc:647] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:56.576790 8732 tablet_copy_client.cc:523] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:56.580309 8732 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:56.582000 8727 tablet_copy_client.cc:308] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.72:41331
I20240503 14:14:56.582969 31223 tablet_copy_service.cc:140] P 530893a961af4b95b6b687fa5ed22f9d: Received BeginTabletCopySession request for tablet bf316cde5631415faecbc1bb26690b79 from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:53445)
I20240503 14:14:56.583124 31223 tablet_copy_service.cc:161] P 530893a961af4b95b6b687fa5ed22f9d: Beginning new tablet copy session on tablet bf316cde5631415faecbc1bb26690b79 from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:53445: session id = fde134d0fa2b4900b9badd2e08fea09d-bf316cde5631415faecbc1bb26690b79
I20240503 14:14:56.585915 31223 tablet_copy_source_session.cc:205] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:56.588330 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf316cde5631415faecbc1bb26690b79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:56.596568 8727 tablet_copy_client.cc:783] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:56.596892 8727 tablet_copy_client.cc:647] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:56.600991 8727 tablet_copy_client.cc:523] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:56.608852 8727 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:56.680757 8732 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:56.681298 8732 tablet_bootstrap.cc:492] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:56.681614 8732 ts_tablet_manager.cc:1367] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.101s user 0.083s sys 0.008s
I20240503 14:14:56.682487 8732 raft_consensus.cc:348] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:56.682718 8732 raft_consensus.cc:729] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:56.682950 8732 consensus_queue.cc:260] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:56.683650 8732 ts_tablet_manager.cc:1398] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:56.686740 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-0225203f2de2498eb448e066ce545a64 received from {username='slave'} at 127.20.240.70:46183
I20240503 14:14:56.686882 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-0225203f2de2498eb448e066ce545a64 on tablet 0225203f2de2498eb448e066ce545a64 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:56.717041 8727 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:56.717464 8727 tablet_bootstrap.cc:492] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:56.717716 8727 ts_tablet_manager.cc:1367] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.075s
I20240503 14:14:56.718470 8727 raft_consensus.cc:348] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.718729 8727 raft_consensus.cc:729] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:56.719003 8727 consensus_queue.cc:260] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.719642 8727 ts_tablet_manager.cc:1398] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:14:56.720052 8744 ts_tablet_manager.cc:902] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:14:56.720605 31223 tablet_copy_service.cc:342] P 530893a961af4b95b6b687fa5ed22f9d: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-bf316cde5631415faecbc1bb26690b79 received from {username='slave'} at 127.20.240.71:53445
I20240503 14:14:56.720732 31223 tablet_copy_service.cc:433] P 530893a961af4b95b6b687fa5ed22f9d: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-bf316cde5631415faecbc1bb26690b79 on tablet bf316cde5631415faecbc1bb26690b79 with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:56.737159 8744 tablet_copy_client.cc:308] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:14:56.737732 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 619951b364d44766a2f3064222ceb738 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:54091)
I20240503 14:14:56.737859 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 619951b364d44766a2f3064222ceb738 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:54091: session id = a4cdf05256a0407c8929ac43eb55d44f-619951b364d44766a2f3064222ceb738
I20240503 14:14:56.740479 30961 tablet_copy_source_session.cc:205] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:56.742141 8744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619951b364d44766a2f3064222ceb738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:56.743762 28304 consensus_queue.cc:237] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.746299 28171 raft_consensus.cc:1261] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.746786 28040 raft_consensus.cc:1261] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:56.748450 28239 consensus_peers.cc:485] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8868e8d0fd6b4a14970df92960ca0b80. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.748744 8512 consensus_queue.cc:1035] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.749259 8512 consensus_queue.cc:1035] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.765291 8744 tablet_copy_client.cc:783] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:56.765527 8744 tablet_copy_client.cc:647] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:56.767796 8744 tablet_copy_client.cc:523] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:56.770501 8744 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:56.773545 8512 raft_consensus.cc:2932] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.774221 28040 raft_consensus.cc:2932] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.785848 27900 catalog_manager.cc:5484] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.787628 28171 raft_consensus.cc:2932] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:56.798918 8744 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:56.799341 8744 tablet_bootstrap.cc:492] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:56.799582 8744 ts_tablet_manager.cc:1367] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20240503 14:14:56.800370 8744 raft_consensus.cc:348] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:56.800565 8744 raft_consensus.cc:729] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:56.800792 8744 consensus_queue.cc:260] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:56.801422 8744 ts_tablet_manager.cc:1398] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:56.802068 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-619951b364d44766a2f3064222ceb738 received from {username='slave'} at 127.20.240.67:54091
I20240503 14:14:56.802177 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-619951b364d44766a2f3064222ceb738 on tablet 619951b364d44766a2f3064222ceb738 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:56.821928 8727 ts_tablet_manager.cc:902] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:56.823097 8744 ts_tablet_manager.cc:902] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:14:56.823644 8727 tablet_copy_client.cc:308] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:56.830340 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 213bc70b5cf14440ab88c754df0b8f7f from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:60297)
I20240503 14:14:56.830482 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 213bc70b5cf14440ab88c754df0b8f7f from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:60297: session id = fde134d0fa2b4900b9badd2e08fea09d-213bc70b5cf14440ab88c754df0b8f7f
I20240503 14:14:56.833081 28454 tablet_copy_source_session.cc:205] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:56.835659 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 213bc70b5cf14440ab88c754df0b8f7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:56.845542 8727 tablet_copy_client.cc:783] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:56.845818 8727 tablet_copy_client.cc:647] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:56.847841 8727 tablet_copy_client.cc:523] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:56.852520 8727 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:56.852603 28040 consensus_queue.cc:237] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:56.856961 8744 tablet_copy_client.cc:308] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:14:56.857645 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet 5be4220923ed4c5388a7a15755cb685d from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:45597)
I20240503 14:14:56.857875 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet 5be4220923ed4c5388a7a15755cb685d from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:45597: session id = a4cdf05256a0407c8929ac43eb55d44f-5be4220923ed4c5388a7a15755cb685d
I20240503 14:14:56.861735 30830 tablet_copy_source_session.cc:205] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:56.863193 8744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be4220923ed4c5388a7a15755cb685d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:14:56.866645 27977 consensus_peers.cc:485] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 859456c340dc47bf8fc600ebe8c4b984. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:56.866834 28304 raft_consensus.cc:1261] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.867447 8715 consensus_queue.cc:1035] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.869155 8744 tablet_copy_client.cc:783] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:56.869355 8744 tablet_copy_client.cc:647] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:56.870091 28434 raft_consensus.cc:1261] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:56.871990 8744 tablet_copy_client.cc:523] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:56.872176 8715 consensus_queue.cc:1035] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:56.875144 8744 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:14:56.876304 8715 raft_consensus.cc:2932] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:56.881493 28433 raft_consensus.cc:2932] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:56.884327 8744 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:56.884886 8744 tablet_bootstrap.cc:492] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:14:56.885224 8744 ts_tablet_manager.cc:1367] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.006s
I20240503 14:14:56.885962 8744 raft_consensus.cc:348] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:56.886284 8744 raft_consensus.cc:729] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:14:56.886555 8744 consensus_queue.cc:260] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:14:56.887176 8744 ts_tablet_manager.cc:1398] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:56.887683 27900 catalog_manager.cc:5484] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:56.888994 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-5be4220923ed4c5388a7a15755cb685d received from {username='slave'} at 127.20.240.67:45597
I20240503 14:14:56.889142 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-5be4220923ed4c5388a7a15755cb685d on tablet 5be4220923ed4c5388a7a15755cb685d with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:14:56.891696 28304 raft_consensus.cc:2932] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:56.926918 8727 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:56.927361 8727 tablet_bootstrap.cc:492] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:56.927636 8727 ts_tablet_manager.cc:1367] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.075s user 0.044s sys 0.001s
I20240503 14:14:56.928429 8727 raft_consensus.cc:348] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.928666 8727 raft_consensus.cc:729] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:56.929006 8727 consensus_queue.cc:260] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:56.929746 8727 ts_tablet_manager.cc:1398] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:14:56.930295 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-213bc70b5cf14440ab88c754df0b8f7f received from {username='slave'} at 127.20.240.71:60297
I20240503 14:14:56.930397 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-213bc70b5cf14440ab88c754df0b8f7f on tablet 213bc70b5cf14440ab88c754df0b8f7f with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:56.946683 31070 raft_consensus.cc:1203] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:56.947554 8788 ts_tablet_manager.cc:902] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:56.971607 8788 tablet_copy_client.cc:308] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:56.972183 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 823dde634aac449ea8cd21d38ecfb22b from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:33841)
I20240503 14:14:56.972313 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 823dde634aac449ea8cd21d38ecfb22b from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:33841: session id = d0dbfb0e43f948c3922926aacc365fc1-823dde634aac449ea8cd21d38ecfb22b
I20240503 14:14:56.974488 28454 tablet_copy_source_session.cc:205] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:56.975709 8788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 823dde634aac449ea8cd21d38ecfb22b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:56.978777 31070 raft_consensus.cc:1203] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:56.983019 8788 tablet_copy_client.cc:783] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:56.983217 8788 tablet_copy_client.cc:647] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:56.985137 8788 tablet_copy_client.cc:523] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:56.988481 8788 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:14:57.004262 8732 ts_tablet_manager.cc:902] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:57.010066 31199 consensus_queue.cc:237] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:57.029431 30941 raft_consensus.cc:1261] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:14:57.029557 31139 consensus_peers.cc:485] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2bb66488d714d39900331f4a257e8b3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.030050 8577 consensus_queue.cc:1035] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:57.031169 30809 raft_consensus.cc:1261] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:14:57.032212 8577 consensus_queue.cc:1035] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:14:57.042814 8732 tablet_copy_client.cc:308] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:57.043646 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet dc9c29880eea4cd9a11eb24eeb3bd228 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:48859)
I20240503 14:14:57.043798 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet dc9c29880eea4cd9a11eb24eeb3bd228 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:48859: session id = 6bbc686598f24f518d7ca6bb2063058d-dc9c29880eea4cd9a11eb24eeb3bd228
I20240503 14:14:57.045933 28063 tablet_copy_source_session.cc:205] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:57.047206 8732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc9c29880eea4cd9a11eb24eeb3bd228. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.047312 8332 raft_consensus.cc:2932] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:57.048480 8788 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:57.048956 8788 tablet_bootstrap.cc:492] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:14:57.049213 8788 ts_tablet_manager.cc:1367] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.061s user 0.050s sys 0.004s
I20240503 14:14:57.049925 8788 raft_consensus.cc:348] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:57.050164 8788 raft_consensus.cc:729] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.050449 8788 consensus_queue.cc:260] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:14:57.051160 8788 ts_tablet_manager.cc:1398] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:57.051789 8732 tablet_copy_client.cc:783] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:57.051959 8732 tablet_copy_client.cc:647] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:57.051988 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-823dde634aac449ea8cd21d38ecfb22b received from {username='slave'} at 127.20.240.69:33841
I20240503 14:14:57.052102 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-823dde634aac449ea8cd21d38ecfb22b on tablet 823dde634aac449ea8cd21d38ecfb22b with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:14:57.058096 30941 raft_consensus.cc:2932] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:57.063035 27894 catalog_manager.cc:5484] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:57.072154 30809 raft_consensus.cc:2932] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:14:57.073432 30941 raft_consensus.cc:1203] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:57.087015 8732 tablet_copy_client.cc:523] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:57.091396 8732 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:57.155647 8732 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:57.156085 8732 tablet_bootstrap.cc:492] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:57.156337 8732 ts_tablet_manager.cc:1367] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.065s user 0.045s sys 0.008s
I20240503 14:14:57.157127 8732 raft_consensus.cc:348] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:57.157410 8732 raft_consensus.cc:729] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.157616 8732 consensus_queue.cc:260] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:57.161283 8732 ts_tablet_manager.cc:1398] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:14:57.161990 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-dc9c29880eea4cd9a11eb24eeb3bd228 received from {username='slave'} at 127.20.240.70:48859
I20240503 14:14:57.162120 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-dc9c29880eea4cd9a11eb24eeb3bd228 on tablet dc9c29880eea4cd9a11eb24eeb3bd228 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:57.171802 28302 consensus_queue.cc:237] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:57.174577 28433 raft_consensus.cc:1261] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:14:57.174649 28239 consensus_peers.cc:485] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 519fed0fc03b4b9788803aeb0bbcfdad. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.174806 28171 raft_consensus.cc:1261] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:57.175140 8512 consensus_queue.cc:1035] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:57.175386 8280 consensus_queue.cc:1035] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:57.178215 8280 raft_consensus.cc:2932] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:57.178824 28171 raft_consensus.cc:2932] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:57.184579 28433 raft_consensus.cc:2932] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:57.189700 27894 catalog_manager.cc:5484] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:57.226090 28302 raft_consensus.cc:1203] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:57.247123 28433 consensus_queue.cc:237] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
W20240503 14:14:57.250104 28369 consensus_peers.cc:485] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edd300244be74e3aa94d7ae35deeb470. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.250133 28304 raft_consensus.cc:1261] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:57.251092 28171 raft_consensus.cc:1261] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:57.251881 8282 consensus_queue.cc:1035] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:57.252656 7344 consensus_queue.cc:1035] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:57.265929 7344 raft_consensus.cc:2932] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:57.267145 28304 raft_consensus.cc:2932] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:57.276553 28172 raft_consensus.cc:2932] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:57.277988 27900 catalog_manager.cc:5484] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:14:57.331749 31069 raft_consensus.cc:1203] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:57.340289 28040 consensus_queue.cc:237] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
W20240503 14:14:57.342475 27977 consensus_peers.cc:485] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25eb1d6c445940d6bd8753a88381bb8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:57.342762 28304 raft_consensus.cc:1261] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:57.342978 28433 raft_consensus.cc:1261] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:14:57.343499 8106 consensus_queue.cc:1035] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:57.343772 8712 consensus_queue.cc:1035] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:14:57.347586 30810 raft_consensus.cc:1203] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:57.347672 8727 ts_tablet_manager.cc:902] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:57.372617 8106 raft_consensus.cc:2932] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:57.378226 28434 raft_consensus.cc:2932] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:57.382409 28304 raft_consensus.cc:1203] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:57.386711 27894 catalog_manager.cc:5484] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:57.389039 28302 raft_consensus.cc:2932] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:14:57.394273 8727 tablet_copy_client.cc:308] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:57.395009 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 8868e8d0fd6b4a14970df92960ca0b80 from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:58557)
I20240503 14:14:57.395150 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 8868e8d0fd6b4a14970df92960ca0b80 from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:58557: session id = fde134d0fa2b4900b9badd2e08fea09d-8868e8d0fd6b4a14970df92960ca0b80
I20240503 14:14:57.397606 28324 tablet_copy_source_session.cc:205] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:57.400123 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8868e8d0fd6b4a14970df92960ca0b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.408697 8727 tablet_copy_client.cc:783] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:57.408990 8727 tablet_copy_client.cc:647] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:57.410758 8727 tablet_copy_client.cc:523] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:57.413961 8727 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:57.442579 8893 ts_tablet_manager.cc:902] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:57.446638 30941 raft_consensus.cc:1203] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:57.465195 8898 ts_tablet_manager.cc:902] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:14:57.466661 8898 tablet_copy_client.cc:308] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.72:41331
I20240503 14:14:57.467257 31223 tablet_copy_service.cc:140] P 530893a961af4b95b6b687fa5ed22f9d: Received BeginTabletCopySession request for tablet d2bb66488d714d39900331f4a257e8b3 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:51783)
I20240503 14:14:57.467406 31223 tablet_copy_service.cc:161] P 530893a961af4b95b6b687fa5ed22f9d: Beginning new tablet copy session on tablet d2bb66488d714d39900331f4a257e8b3 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:51783: session id = 793a0d5dd1634c5faaf0e1e4851e8294-d2bb66488d714d39900331f4a257e8b3
I20240503 14:14:57.468554 8893 tablet_copy_client.cc:308] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:57.469257 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 859456c340dc47bf8fc600ebe8c4b984 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:32803)
I20240503 14:14:57.469396 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 859456c340dc47bf8fc600ebe8c4b984 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:32803: session id = 530893a961af4b95b6b687fa5ed22f9d-859456c340dc47bf8fc600ebe8c4b984
I20240503 14:14:57.470485 31223 tablet_copy_source_session.cc:205] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:57.471819 28063 tablet_copy_source_session.cc:205] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:57.472970 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 859456c340dc47bf8fc600ebe8c4b984. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.473527 8898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2bb66488d714d39900331f4a257e8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.474462 8727 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:57.474925 8727 tablet_bootstrap.cc:492] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:57.475198 8727 ts_tablet_manager.cc:1367] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.061s user 0.045s sys 0.004s
I20240503 14:14:57.476006 8727 raft_consensus.cc:348] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:57.476225 8727 raft_consensus.cc:729] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.476462 8727 consensus_queue.cc:260] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:57.478753 8727 ts_tablet_manager.cc:1398] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:14:57.480975 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-8868e8d0fd6b4a14970df92960ca0b80 received from {username='slave'} at 127.20.240.71:58557
I20240503 14:14:57.481094 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-8868e8d0fd6b4a14970df92960ca0b80 on tablet 8868e8d0fd6b4a14970df92960ca0b80 with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:57.481323 8893 tablet_copy_client.cc:783] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:57.481540 8893 tablet_copy_client.cc:647] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:57.483059 8893 tablet_copy_client.cc:523] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:57.486271 8893 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:14:57.490765 8898 tablet_copy_client.cc:783] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:57.491016 8898 tablet_copy_client.cc:647] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:57.492339 8898 tablet_copy_client.cc:523] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:57.495870 8898 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:14:57.520517 8898 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:57.521106 8898 tablet_bootstrap.cc:492] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:14:57.521410 8898 ts_tablet_manager.cc:1367] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.013s
I20240503 14:14:57.522315 8898 raft_consensus.cc:348] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:57.522542 8898 raft_consensus.cc:729] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.522842 8898 consensus_queue.cc:260] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:14:57.523562 8898 ts_tablet_manager.cc:1398] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:14:57.524943 31223 tablet_copy_service.cc:342] P 530893a961af4b95b6b687fa5ed22f9d: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-d2bb66488d714d39900331f4a257e8b3 received from {username='slave'} at 127.20.240.65:51783
I20240503 14:14:57.525076 31223 tablet_copy_service.cc:433] P 530893a961af4b95b6b687fa5ed22f9d: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-d2bb66488d714d39900331f4a257e8b3 on tablet d2bb66488d714d39900331f4a257e8b3 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:14:57.540155 8893 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:57.540588 8893 tablet_bootstrap.cc:492] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:14:57.540859 8893 ts_tablet_manager.cc:1367] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.055s user 0.040s sys 0.004s
I20240503 14:14:57.541626 8893 raft_consensus.cc:348] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:57.541893 8893 raft_consensus.cc:729] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.542140 8893 consensus_queue.cc:260] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:57.544502 8893 ts_tablet_manager.cc:1398] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:14:57.545085 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-859456c340dc47bf8fc600ebe8c4b984 received from {username='slave'} at 127.20.240.72:32803
I20240503 14:14:57.545233 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-859456c340dc47bf8fc600ebe8c4b984 on tablet 859456c340dc47bf8fc600ebe8c4b984 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:14:57.610657 8732 ts_tablet_manager.cc:902] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:14:57.622912 8732 tablet_copy_client.cc:308] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:14:57.627847 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 519fed0fc03b4b9788803aeb0bbcfdad from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:57239)
I20240503 14:14:57.627982 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 519fed0fc03b4b9788803aeb0bbcfdad from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:57239: session id = 6bbc686598f24f518d7ca6bb2063058d-519fed0fc03b4b9788803aeb0bbcfdad
I20240503 14:14:57.630718 28324 tablet_copy_source_session.cc:205] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:57.632540 8732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 519fed0fc03b4b9788803aeb0bbcfdad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.638708 8732 tablet_copy_client.cc:783] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:57.639041 8732 tablet_copy_client.cc:647] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:57.641690 8732 tablet_copy_client.cc:523] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:57.645426 8732 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:14:57.712970 8727 ts_tablet_manager.cc:902] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:14:57.723719 8732 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:57.729140 8732 tablet_bootstrap.cc:492] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:14:57.729916 8732 ts_tablet_manager.cc:1367] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.085s user 0.057s sys 0.008s
I20240503 14:14:57.730989 8732 raft_consensus.cc:348] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:57.732107 8732 raft_consensus.cc:729] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.732446 8732 consensus_queue.cc:260] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:57.734894 8732 ts_tablet_manager.cc:1398] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:14:57.738008 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-519fed0fc03b4b9788803aeb0bbcfdad received from {username='slave'} at 127.20.240.70:57239
I20240503 14:14:57.738158 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-519fed0fc03b4b9788803aeb0bbcfdad on tablet 519fed0fc03b4b9788803aeb0bbcfdad with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:14:57.758462 8727 tablet_copy_client.cc:308] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:14:57.759042 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet edd300244be74e3aa94d7ae35deeb470 from peer fde134d0fa2b4900b9badd2e08fea09d ({username='slave'} at 127.20.240.71:60297)
I20240503 14:14:57.759194 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet edd300244be74e3aa94d7ae35deeb470 from peer fde134d0fa2b4900b9badd2e08fea09d at {username='slave'} at 127.20.240.71:60297: session id = fde134d0fa2b4900b9badd2e08fea09d-edd300244be74e3aa94d7ae35deeb470
I20240503 14:14:57.761533 28454 tablet_copy_source_session.cc:205] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:57.762612 8727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edd300244be74e3aa94d7ae35deeb470. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.770684 8727 tablet_copy_client.cc:783] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:57.770902 8727 tablet_copy_client.cc:647] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:57.787046 8727 tablet_copy_client.cc:523] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:57.790985 8727 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap starting.
I20240503 14:14:57.852491 8727 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:57.852980 8727 tablet_bootstrap.cc:492] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: Bootstrap complete.
I20240503 14:14:57.853302 8727 ts_tablet_manager.cc:1367] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: Time spent bootstrapping tablet: real 0.062s user 0.046s sys 0.005s
I20240503 14:14:57.854070 8727 raft_consensus.cc:348] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:57.854375 8727 raft_consensus.cc:729] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Initialized, Role: FOLLOWER
I20240503 14:14:57.854724 8727 consensus_queue.cc:260] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:14:57.855494 8727 ts_tablet_manager.cc:1398] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:14:57.856344 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session fde134d0fa2b4900b9badd2e08fea09d-edd300244be74e3aa94d7ae35deeb470 received from {username='slave'} at 127.20.240.71:60297
I20240503 14:14:57.856457 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session fde134d0fa2b4900b9badd2e08fea09d-edd300244be74e3aa94d7ae35deeb470 on tablet edd300244be74e3aa94d7ae35deeb470 with peer fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:14:57.875950 8893 ts_tablet_manager.cc:902] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:14:57.886059 31070 raft_consensus.cc:1203] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:57.919445 8893 tablet_copy_client.cc:308] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:14:57.920097 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 25eb1d6c445940d6bd8753a88381bb8c from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:32803)
I20240503 14:14:57.920244 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 25eb1d6c445940d6bd8753a88381bb8c from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:32803: session id = 530893a961af4b95b6b687fa5ed22f9d-25eb1d6c445940d6bd8753a88381bb8c
I20240503 14:14:57.923146 28063 tablet_copy_source_session.cc:205] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:57.924331 8893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25eb1d6c445940d6bd8753a88381bb8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:57.933001 8893 tablet_copy_client.cc:783] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:57.933271 8893 tablet_copy_client.cc:647] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:57.935142 8893 tablet_copy_client.cc:523] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:57.941102 8893 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:14:58.014256 8893 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:58.014864 8893 tablet_bootstrap.cc:492] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:14:58.046030 31202 raft_consensus.cc:1203] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:58.055534 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:58.056722 8893 ts_tablet_manager.cc:1367] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.116s user 0.066s sys 0.009s
I20240503 14:14:58.061520 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:58.059481 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:58.057461 8893 raft_consensus.cc:348] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:58.064082 8893 raft_consensus.cc:729] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:14:58.064350 8893 consensus_queue.cc:260] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:58.064872 8893 ts_tablet_manager.cc:1398] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:14:58.065687 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-25eb1d6c445940d6bd8753a88381bb8c received from {username='slave'} at 127.20.240.72:32803
I20240503 14:14:58.065811 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-25eb1d6c445940d6bd8753a88381bb8c on tablet 25eb1d6c445940d6bd8753a88381bb8c with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:14:58.073784 28041 raft_consensus.cc:1203] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:14:58.106772 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:58.107331 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:58.121199 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:58.129218 31070 raft_consensus.cc:1203] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:58.137333 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:58.175849 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:58.189604 30941 raft_consensus.cc:1203] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:58.272019 31199 raft_consensus.cc:1203] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:14:58.631608 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:58.637771 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:58.643399 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:58.661857 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:58.694554 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:58.705199 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 14:14:58.773752 28110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.67:43889, user_credentials={real_user=slave}} blocked reactor thread for 114122us
W20240503 14:14:58.794639 28107 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 87969.1us
I20240503 14:14:58.803201 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:58.834709 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:14:58.980412 28303 consensus_queue.cc:237] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:14:58.982461 28172 raft_consensus.cc:1261] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:58.983016 9056 consensus_queue.cc:1035] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:58.983232 30941 raft_consensus.cc:1261] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:58.984167 9055 consensus_queue.cc:1035] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:14:58.985455 9056 raft_consensus.cc:2932] T 519fed0fc03b4b9788803aeb0bbcfdad P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:58.986075 28172 raft_consensus.cc:2932] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:58.990465 30941 raft_consensus.cc:2932] T 519fed0fc03b4b9788803aeb0bbcfdad P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:58.992180 27897 catalog_manager.cc:5484] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:14:59.000526 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 519fed0fc03b4b9788803aeb0bbcfdad with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58086
I20240503 14:14:59.001152 9081 tablet_replica.cc:330] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:14:59.001454 9081 raft_consensus.cc:2229] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:14:59.001766 9081 raft_consensus.cc:2258] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:14:59.003959 9081 ts_tablet_manager.cc:1875] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:14:59.009943 9081 ts_tablet_manager.cc:1888] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:14:59.010068 9081 log.cc:1196] T 519fed0fc03b4b9788803aeb0bbcfdad P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/519fed0fc03b4b9788803aeb0bbcfdad
I20240503 14:14:59.010794 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 519fed0fc03b4b9788803aeb0bbcfdad (table rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]) successfully deleted
I20240503 14:14:59.270711 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:14:59.349570 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:59.384253 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:14:59.385529 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:59.389066 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:14:59.403664 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:14:59.405781 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:14:59.458743 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:14:59.506675 30877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 49009.2us
I20240503 14:14:59.592942 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(8ddd3f269b35481baa313da70ee987de): perf score=0.033352
I20240503 14:14:59.629741 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(8ddd3f269b35481baa313da70ee987de): perf score=0.033358
I20240503 14:14:59.700366 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(8ddd3f269b35481baa313da70ee987de): perf score=0.033378
I20240503 14:14:59.732235 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(ac08c6dba4dc4e90ab11a155ec6ecdfd): perf score=0.033387
I20240503 14:14:59.796769 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(8ddd3f269b35481baa313da70ee987de) complete. Timing: real 0.201s user 0.056s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":347,"dirs.queue_time_us":382,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":896,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":921,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":19894,"threads_started":3,"wal-append.queue_time_us":16044}
I20240503 14:14:59.799938 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(8ddd3f269b35481baa313da70ee987de): 4838 bytes on disk
I20240503 14:14:59.803645 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(8ddd3f269b35481baa313da70ee987de) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 14:14:59.933718 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(ac08c6dba4dc4e90ab11a155ec6ecdfd) complete. Timing: real 0.199s user 0.061s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":10888,"dirs.queue_time_us":3432,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1035,"drs_written":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3577,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":39398,"threads_started":3,"wal-append.queue_time_us":15489}
I20240503 14:14:59.937373 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(d7b192797b6c40d4853e6ae4583ef3ca): perf score=0.033441
I20240503 14:14:59.975919 28041 consensus_queue.cc:237] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:14:59.990773 31202 raft_consensus.cc:1261] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:14:59.992501 9031 consensus_queue.cc:1035] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:00.009128 28433 raft_consensus.cc:1261] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:00.009706 9031 consensus_queue.cc:1035] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:00.011361 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(8ddd3f269b35481baa313da70ee987de) complete. Timing: real 0.381s user 0.081s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":15041,"dirs.queue_time_us":13289,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":10374,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2526,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":44752,"threads_started":3,"wal-append.queue_time_us":16413}
I20240503 14:15:00.016631 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(8ddd3f269b35481baa313da70ee987de): 4838 bytes on disk
I20240503 14:15:00.017489 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(8ddd3f269b35481baa313da70ee987de) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 14:15:00.023610 9031 raft_consensus.cc:2932] T 859456c340dc47bf8fc600ebe8c4b984 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:00.028319 31199 raft_consensus.cc:2932] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:00.029623 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(d7b192797b6c40d4853e6ae4583ef3ca): perf score=0.033464
I20240503 14:15:00.037879 27900 catalog_manager.cc:5484] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:00.057912 28434 raft_consensus.cc:2932] T 859456c340dc47bf8fc600ebe8c4b984 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:00.112977 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 859456c340dc47bf8fc600ebe8c4b984 with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43626
I20240503 14:15:00.132169 9185 tablet_replica.cc:330] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:15:00.134581 9185 raft_consensus.cc:2229] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:00.135568 9185 raft_consensus.cc:2258] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:00.136325 9185 ts_tablet_manager.cc:1875] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:00.143908 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(d7b192797b6c40d4853e6ae4583ef3ca) complete. Timing: real 0.202s user 0.037s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1502,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":958,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12121,"threads_started":1,"wal-append.queue_time_us":12032}
I20240503 14:15:00.155328 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(ac08c6dba4dc4e90ab11a155ec6ecdfd): 4923 bytes on disk
I20240503 14:15:00.162669 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(ac08c6dba4dc4e90ab11a155ec6ecdfd) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 14:15:00.163504 9185 ts_tablet_manager.cc:1888] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:00.163684 9185 log.cc:1196] T 859456c340dc47bf8fc600ebe8c4b984 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/859456c340dc47bf8fc600ebe8c4b984
I20240503 14:15:00.164592 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 859456c340dc47bf8fc600ebe8c4b984 (table rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]) successfully deleted
I20240503 14:15:00.170688 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(d7b192797b6c40d4853e6ae4583ef3ca): 4583 bytes on disk
I20240503 14:15:00.171288 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(d7b192797b6c40d4853e6ae4583ef3ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 14:15:00.192548 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(8ddd3f269b35481baa313da70ee987de) complete. Timing: real 0.491s user 0.079s sys 0.025s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":18678,"dirs.queue_time_us":52285,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":8355,"drs_written":1,"lbm_read_time_us":1398,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":2622,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":91462,"threads_started":3,"wal-append.queue_time_us":12190}
I20240503 14:15:00.200461 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(8ddd3f269b35481baa313da70ee987de): 4838 bytes on disk
I20240503 14:15:00.206876 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(8ddd3f269b35481baa313da70ee987de) complete. Timing: real 0.005s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
I20240503 14:15:00.210677 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(ac08c6dba4dc4e90ab11a155ec6ecdfd): perf score=0.033518
I20240503 14:15:00.324095 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(d7b192797b6c40d4853e6ae4583ef3ca) complete. Timing: real 0.292s user 0.038s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1790,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1342,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":18939,"threads_started":1,"wal-append.queue_time_us":15824}
I20240503 14:15:00.332412 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(d7b192797b6c40d4853e6ae4583ef3ca): 4583 bytes on disk
I20240503 14:15:00.333045 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(d7b192797b6c40d4853e6ae4583ef3ca) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 14:15:00.463837 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(ac08c6dba4dc4e90ab11a155ec6ecdfd) complete. Timing: real 0.252s user 0.044s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":842,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":44088,"threads_started":1,"wal-append.queue_time_us":44257}
I20240503 14:15:00.470485 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(ac08c6dba4dc4e90ab11a155ec6ecdfd): 4923 bytes on disk
I20240503 14:15:00.471544 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(ac08c6dba4dc4e90ab11a155ec6ecdfd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240503 14:15:01.229000 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 14:15:01.280918 28368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 79388us
I20240503 14:15:01.300290 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:01.308013 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:01.316495 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:01.322667 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:01.365844 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:01.381795 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:01.457561 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:01.576495 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(400ce95594a743108bb4ee1e64458956): perf score=0.033344
I20240503 14:15:01.605829 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(519fed0fc03b4b9788803aeb0bbcfdad): perf score=0.033351
I20240503 14:15:01.689891 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(400ce95594a743108bb4ee1e64458956): perf score=0.033372
I20240503 14:15:01.753115 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(88f88e7eb7434450af76714256729887): perf score=0.033389
I20240503 14:15:01.766347 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(519fed0fc03b4b9788803aeb0bbcfdad) complete. Timing: real 0.160s user 0.066s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":31314,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":3977,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1272,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":46371,"threads_started":2,"wal-append.queue_time_us":13922}
I20240503 14:15:01.770023 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(400ce95594a743108bb4ee1e64458956): perf score=0.033393
I20240503 14:15:01.820501 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(400ce95594a743108bb4ee1e64458956) complete. Timing: real 0.244s user 0.059s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":325666,"compiler_manager_pool.run_cpu_time_us":50,"compiler_manager_pool.run_wall_time_us":50,"dirs.queue_time_us":10980,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":23458,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":984,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":22171,"threads_started":2,"wal-append.queue_time_us":10940}
I20240503 14:15:01.824223 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(400ce95594a743108bb4ee1e64458956): 4583 bytes on disk
I20240503 14:15:01.824810 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(88f88e7eb7434450af76714256729887) complete. Timing: real 0.071s user 0.039s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1280,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1507,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13641,"threads_started":2,"wal-append.queue_time_us":6239}
I20240503 14:15:01.825554 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(400ce95594a743108bb4ee1e64458956) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 14:15:01.827669 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887): 4924 bytes on disk
I20240503 14:15:01.828879 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 14:15:01.831390 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(88f88e7eb7434450af76714256729887): perf score=0.033412
I20240503 14:15:01.832070 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(519fed0fc03b4b9788803aeb0bbcfdad): perf score=0.033404
I20240503 14:15:01.848140 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(400ce95594a743108bb4ee1e64458956) complete. Timing: real 0.078s user 0.039s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":18051,"compiler_manager_pool.run_cpu_time_us":27,"compiler_manager_pool.run_wall_time_us":27,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1385,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":725,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7658,"threads_started":1,"wal-append.queue_time_us":7422}
I20240503 14:15:01.853241 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(88f88e7eb7434450af76714256729887): perf score=0.033410
I20240503 14:15:01.882769 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(400ce95594a743108bb4ee1e64458956) complete. Timing: real 0.192s user 0.052s sys 0.032s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":7579,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":4292,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":820,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":49791,"threads_started":2,"wal-append.queue_time_us":38193}
I20240503 14:15:01.885802 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(400ce95594a743108bb4ee1e64458956): 4583 bytes on disk
I20240503 14:15:01.887282 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(400ce95594a743108bb4ee1e64458956) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240503 14:15:01.947474 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(88f88e7eb7434450af76714256729887) complete. Timing: real 0.116s user 0.042s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1666,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":925,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":11544,"threads_started":1,"wal-append.queue_time_us":11451}
I20240503 14:15:01.950277 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887): 4924 bytes on disk
I20240503 14:15:01.950874 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 14:15:01.956209 28302 consensus_queue.cc:237] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:01.967463 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(519fed0fc03b4b9788803aeb0bbcfdad) complete. Timing: real 0.135s user 0.047s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1105,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":20220,"threads_started":1,"wal-append.queue_time_us":12232}
I20240503 14:15:01.970321 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(519fed0fc03b4b9788803aeb0bbcfdad): 4838 bytes on disk
I20240503 14:15:01.972560 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(519fed0fc03b4b9788803aeb0bbcfdad) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1745,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240503 14:15:01.974188 31070 raft_consensus.cc:1261] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:01.975898 9140 consensus_queue.cc:1035] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:01.976712 28040 raft_consensus.cc:1261] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:01.977308 9140 consensus_queue.cc:1035] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:01.981079 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(88f88e7eb7434450af76714256729887) complete. Timing: real 0.119s user 0.047s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1210,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":2367,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":13119,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6583,"threads_started":1,"wal-append.queue_time_us":6501}
I20240503 14:15:01.985090 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887): 4924 bytes on disk
I20240503 14:15:01.985751 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240503 14:15:01.989478 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(519fed0fc03b4b9788803aeb0bbcfdad): 4838 bytes on disk
I20240503 14:15:01.990049 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(519fed0fc03b4b9788803aeb0bbcfdad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240503 14:15:01.994349 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(400ce95594a743108bb4ee1e64458956): 4583 bytes on disk
I20240503 14:15:01.994968 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(400ce95594a743108bb4ee1e64458956) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":136,"lbm_reads_lt_1ms":4}
I20240503 14:15:02.003350 9055 raft_consensus.cc:2932] T 8868e8d0fd6b4a14970df92960ca0b80 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:02.004355 28041 raft_consensus.cc:2932] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:02.012840 27897 catalog_manager.cc:5484] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:02.034137 31070 raft_consensus.cc:2932] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:02.050854 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 8868e8d0fd6b4a14970df92960ca0b80 with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:15:02.055011 9302 tablet_replica.cc:330] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:15:02.061328 9302 raft_consensus.cc:2229] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:02.062168 9302 raft_consensus.cc:2258] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:02.063514 9302 ts_tablet_manager.cc:1875] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:02.081797 9302 ts_tablet_manager.cc:1888] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:02.081990 9302 log.cc:1196] T 8868e8d0fd6b4a14970df92960ca0b80 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/8868e8d0fd6b4a14970df92960ca0b80
I20240503 14:15:02.083277 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 8868e8d0fd6b4a14970df92960ca0b80 (table rebalance_test_table_11 [id=6639b1cb53f44be0b930873e71075e93]) successfully deleted
I20240503 14:15:02.417104 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:02.430717 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:02.433192 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:02.443629 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:02.452186 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:02.467124 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:02.471149 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:02.482995 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:02.720597 28041 consensus_queue.cc:237] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:02.722332 30939 raft_consensus.cc:1261] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:02.722414 28172 raft_consensus.cc:1261] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:02.722908 9257 consensus_queue.cc:1035] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:02.723168 9031 consensus_queue.cc:1035] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:02.725441 9257 raft_consensus.cc:2932] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:02.725982 28172 raft_consensus.cc:2932] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:02.726547 30939 raft_consensus.cc:2932] T dc9c29880eea4cd9a11eb24eeb3bd228 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:02.732124 27902 catalog_manager.cc:5484] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:02.744683 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet dc9c29880eea4cd9a11eb24eeb3bd228 with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58086
I20240503 14:15:02.745303 9395 tablet_replica.cc:330] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:15:02.745613 9395 raft_consensus.cc:2229] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:02.745954 9395 raft_consensus.cc:2258] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:02.746690 9395 ts_tablet_manager.cc:1875] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:02.752259 9395 ts_tablet_manager.cc:1888] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:02.752440 9395 log.cc:1196] T dc9c29880eea4cd9a11eb24eeb3bd228 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/dc9c29880eea4cd9a11eb24eeb3bd228
I20240503 14:15:02.753104 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet dc9c29880eea4cd9a11eb24eeb3bd228 (table rebalance_test_table_15 [id=f1a0906eb4d644e88b1eb9ef75dc7b19]) successfully deleted
I20240503 14:15:03.005657 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:03.056867 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:03.084833 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:03.091967 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:03.153527 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:03.154055 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:03.159163 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:03.168735 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:03.323346 28432 consensus_queue.cc:237] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:03.325048 30807 raft_consensus.cc:1261] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:03.325193 28303 raft_consensus.cc:1261] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:03.325544 9252 consensus_queue.cc:1035] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:03.325798 7344 consensus_queue.cc:1035] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:03.328320 9252 raft_consensus.cc:2932] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:03.328963 30807 raft_consensus.cc:2932] T 823dde634aac449ea8cd21d38ecfb22b P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:03.335505 27900 catalog_manager.cc:5484] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:03.343984 28302 raft_consensus.cc:2932] T 823dde634aac449ea8cd21d38ecfb22b P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:03.354367 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 823dde634aac449ea8cd21d38ecfb22b with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52616
I20240503 14:15:03.358675 9474 tablet_replica.cc:330] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:15:03.358867 9474 raft_consensus.cc:2229] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:03.359169 9474 raft_consensus.cc:2258] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:03.359905 9474 ts_tablet_manager.cc:1875] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:03.370077 9474 ts_tablet_manager.cc:1888] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:03.370294 9474 log.cc:1196] T 823dde634aac449ea8cd21d38ecfb22b P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/823dde634aac449ea8cd21d38ecfb22b
I20240503 14:15:03.371480 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 823dde634aac449ea8cd21d38ecfb22b (table rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]) successfully deleted
I20240503 14:15:03.716192 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:03.777532 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:03.780658 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:03.793642 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:03.860577 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:03.861655 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:03.864946 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:03.899935 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:04.063812 30810 consensus_queue.cc:237] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:04.065645 28303 raft_consensus.cc:1261] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:04.066128 9545 consensus_queue.cc:1035] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:04.068537 31069 raft_consensus.cc:1261] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:04.068465 9432 raft_consensus.cc:2932] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:04.070274 28303 raft_consensus.cc:2932] T 5be4220923ed4c5388a7a15755cb685d P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:04.075150 27899 catalog_manager.cc:5484] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:04.078653 9545 consensus_queue.cc:1035] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:04.098104 31069 raft_consensus.cc:2932] T 5be4220923ed4c5388a7a15755cb685d P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:04.110082 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be4220923ed4c5388a7a15755cb685d with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:15:04.112926 9577 tablet_replica.cc:330] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:15:04.113168 9577 raft_consensus.cc:2229] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:04.113390 9577 raft_consensus.cc:2258] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:04.114590 9577 ts_tablet_manager.cc:1875] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:04.122378 9577 ts_tablet_manager.cc:1888] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:04.122505 9577 log.cc:1196] T 5be4220923ed4c5388a7a15755cb685d P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/5be4220923ed4c5388a7a15755cb685d
I20240503 14:15:04.123273 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 5be4220923ed4c5388a7a15755cb685d (table rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]) successfully deleted
I20240503 14:15:04.446072 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:04.476608 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 14:15:04.493767 31139 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.67:43889, user_credentials={real_user=slave}} blocked reactor thread for 67494.8us
I20240503 14:15:04.498528 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
W20240503 14:15:04.535558 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 61607.1us
I20240503 14:15:04.569895 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:04.610265 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:04.614363 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:04.621541 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:15:04.648325 30875 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.72:41331, user_credentials={real_user=slave}} blocked reactor thread for 110575us
I20240503 14:15:04.654826 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:04.672140 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(609db8c3f3b245eaa31c82915b61188f): perf score=0.033361
I20240503 14:15:04.741009 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(a816d34a63b945b08eec771d9e7c2d55): perf score=0.033375
I20240503 14:15:04.753743 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(a816d34a63b945b08eec771d9e7c2d55): perf score=0.033378
I20240503 14:15:04.767503 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(609db8c3f3b245eaa31c82915b61188f) complete. Timing: real 0.095s user 0.046s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6972,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":9390,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13565,"threads_started":2,"wal-append.queue_time_us":6371}
I20240503 14:15:04.770485 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(609db8c3f3b245eaa31c82915b61188f): 4924 bytes on disk
I20240503 14:15:04.772091 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(609db8c3f3b245eaa31c82915b61188f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 14:15:04.774425 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(49d89642d5364479877c7ba4700fdcee): perf score=0.033385
I20240503 14:15:04.780066 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(609db8c3f3b245eaa31c82915b61188f): perf score=0.033388
I20240503 14:15:04.858588 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(a816d34a63b945b08eec771d9e7c2d55) complete. Timing: real 0.117s user 0.035s sys 0.017s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":15508,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":5005,"drs_written":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4,"lbm_write_time_us":848,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":19551,"threads_started":2,"wal-append.queue_time_us":9428}
I20240503 14:15:04.865052 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(a816d34a63b945b08eec771d9e7c2d55): 4839 bytes on disk
I20240503 14:15:04.866441 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(a816d34a63b945b08eec771d9e7c2d55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":153,"lbm_reads_lt_1ms":4}
I20240503 14:15:04.873076 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(a816d34a63b945b08eec771d9e7c2d55) complete. Timing: real 0.119s user 0.044s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":8916,"dirs.run_cpu_time_us":149,"dirs.run_wall_time_us":7042,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":977,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16916,"threads_started":2,"wal-append.queue_time_us":7809}
I20240503 14:15:04.875731 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(49d89642d5364479877c7ba4700fdcee): perf score=0.033410
I20240503 14:15:04.884279 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(49d89642d5364479877c7ba4700fdcee) complete. Timing: real 0.107s user 0.057s sys 0.043s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1256,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":731,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":35872,"threads_started":1,"wal-append.queue_time_us":35952}
I20240503 14:15:04.887202 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(49d89642d5364479877c7ba4700fdcee): 4583 bytes on disk
I20240503 14:15:04.887903 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(49d89642d5364479877c7ba4700fdcee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 14:15:04.961158 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(609db8c3f3b245eaa31c82915b61188f) complete. Timing: real 0.181s user 0.080s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":279,"dirs.run_cpu_time_us":137,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":793,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":53349,"threads_started":2,"wal-append.queue_time_us":43653}
I20240503 14:15:04.964543 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(609db8c3f3b245eaa31c82915b61188f): 4924 bytes on disk
I20240503 14:15:04.965123 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(609db8c3f3b245eaa31c82915b61188f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 14:15:04.995828 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(49d89642d5364479877c7ba4700fdcee) complete. Timing: real 0.119s user 0.055s sys 0.031s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1473,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19048,"threads_started":1,"wal-append.queue_time_us":19128}
I20240503 14:15:04.998137 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(a816d34a63b945b08eec771d9e7c2d55): 4839 bytes on disk
I20240503 14:15:04.998699 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(a816d34a63b945b08eec771d9e7c2d55) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 14:15:05.000177 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(49d89642d5364479877c7ba4700fdcee): 4583 bytes on disk
I20240503 14:15:05.000638 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(49d89642d5364479877c7ba4700fdcee) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240503 14:15:05.035542 28172 consensus_queue.cc:237] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:05.037281 30939 raft_consensus.cc:1261] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:05.037313 28040 raft_consensus.cc:1261] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:05.037757 9612 consensus_queue.cc:1035] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:05.038000 9449 consensus_queue.cc:1035] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:05.040231 9612 raft_consensus.cc:2932] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:05.040760 28040 raft_consensus.cc:2932] T 0225203f2de2498eb448e066ce545a64 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:05.047276 27894 catalog_manager.cc:5484] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:05.050645 30939 raft_consensus.cc:2932] T 0225203f2de2498eb448e066ce545a64 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:05.055433 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 0225203f2de2498eb448e066ce545a64 with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58086
I20240503 14:15:05.056077 9675 tablet_replica.cc:330] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:15:05.056290 9675 raft_consensus.cc:2229] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:05.056602 9675 raft_consensus.cc:2258] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:05.057281 9675 ts_tablet_manager.cc:1875] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:05.064491 9675 ts_tablet_manager.cc:1888] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:05.064674 9675 log.cc:1196] T 0225203f2de2498eb448e066ce545a64 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/0225203f2de2498eb448e066ce545a64
I20240503 14:15:05.065483 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 0225203f2de2498eb448e066ce545a64 (table rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]) successfully deleted
I20240503 14:15:05.377434 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:05.380656 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:05.382701 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:05.394271 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:05.453809 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:05.456784 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:05.507308 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:05.512722 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:05.699498 31202 consensus_queue.cc:237] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:05.701741 28040 raft_consensus.cc:1261] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:05.701745 30939 raft_consensus.cc:1261] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:05.702337 9625 consensus_queue.cc:1035] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:05.702863 9141 consensus_queue.cc:1035] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:05.704867 9625 raft_consensus.cc:2932] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:05.705372 28040 raft_consensus.cc:2932] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:05.710795 30939 raft_consensus.cc:2932] T d2bb66488d714d39900331f4a257e8b3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:05.722698 27900 catalog_manager.cc:5484] T d2bb66488d714d39900331f4a257e8b3 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:05.732327 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet d2bb66488d714d39900331f4a257e8b3 with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:15:05.733667 9761 tablet_replica.cc:330] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:15:05.733906 9761 raft_consensus.cc:2229] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:05.734189 9761 raft_consensus.cc:2258] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:05.734974 9761 ts_tablet_manager.cc:1875] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:05.742939 9761 ts_tablet_manager.cc:1888] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:05.743053 9761 log.cc:1196] T d2bb66488d714d39900331f4a257e8b3 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/d2bb66488d714d39900331f4a257e8b3
I20240503 14:15:05.743697 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet d2bb66488d714d39900331f4a257e8b3 (table rebalancer_extra_table_30 [id=4d8ccd3a7c63497282c865563c590b02]) successfully deleted
I20240503 14:15:05.983024 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:05.984299 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:05.993615 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:06.003257 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:06.056658 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:06.090516 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:06.151222 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:06.175060 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:06.290681 30939 consensus_queue.cc:237] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:06.292583 31072 raft_consensus.cc:1261] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:06.292722 28434 raft_consensus.cc:1261] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:06.293066 9799 consensus_queue.cc:1035] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:06.293258 9144 consensus_queue.cc:1035] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:06.295457 9144 raft_consensus.cc:2932] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:06.296108 28434 raft_consensus.cc:2932] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:06.303998 27895 catalog_manager.cc:5484] T ee914d743a2c46418f79728f3d32d1c5 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:06.319707 31072 raft_consensus.cc:2932] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:06.331054 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet ee914d743a2c46418f79728f3d32d1c5 with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:15:06.334800 9848 tablet_replica.cc:330] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:15:06.335124 9848 raft_consensus.cc:2229] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:06.335587 9848 raft_consensus.cc:2258] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:06.336186 9848 ts_tablet_manager.cc:1875] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:06.342017 9848 ts_tablet_manager.cc:1888] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:06.342150 9848 log.cc:1196] T ee914d743a2c46418f79728f3d32d1c5 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/ee914d743a2c46418f79728f3d32d1c5
I20240503 14:15:06.342921 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet ee914d743a2c46418f79728f3d32d1c5 (table rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]) successfully deleted
I20240503 14:15:06.617758 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:06.628582 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:06.633193 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:06.697291 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:06.713778 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:06.769876 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:15:06.819725 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 167957us
I20240503 14:15:06.831813 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:06.852407 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:06.993284 30939 consensus_queue.cc:237] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:06.995409 30807 raft_consensus.cc:1261] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:06.995443 28302 raft_consensus.cc:1261] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:06.995867 9799 consensus_queue.cc:1035] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:06.996138 9144 consensus_queue.cc:1035] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:06.998592 9799 raft_consensus.cc:2932] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:06.999157 30807 raft_consensus.cc:2932] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:07.004453 28302 raft_consensus.cc:2932] T 619951b364d44766a2f3064222ceb738 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:07.005175 27900 catalog_manager.cc:5484] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:07.012190 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(f133ebe695a74f86be1a37ae661719ec): perf score=0.033347
I20240503 14:15:07.014693 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 619951b364d44766a2f3064222ceb738 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:15:07.016055 9935 tablet_replica.cc:330] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:15:07.016659 9935 raft_consensus.cc:2229] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:07.017000 9935 raft_consensus.cc:2258] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:07.017987 9935 ts_tablet_manager.cc:1875] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:07.026103 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(1bb01c0be7da4fe4b9c7c7160df7debc): perf score=0.033348
I20240503 14:15:07.035892 9935 ts_tablet_manager.cc:1888] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:07.036069 9935 log.cc:1196] T 619951b364d44766a2f3064222ceb738 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/619951b364d44766a2f3064222ceb738
I20240503 14:15:07.036825 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 619951b364d44766a2f3064222ceb738 (table rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]) successfully deleted
I20240503 14:15:07.110389 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(f133ebe695a74f86be1a37ae661719ec) complete. Timing: real 0.098s user 0.054s sys 0.028s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1412,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":2596,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1110,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14711,"threads_started":2,"wal-append.queue_time_us":12939}
I20240503 14:15:07.112987 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(1bb01c0be7da4fe4b9c7c7160df7debc): perf score=0.033371
I20240503 14:15:07.139034 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(1bb01c0be7da4fe4b9c7c7160df7debc) complete. Timing: real 0.112s user 0.058s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1722,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":9339,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1204,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6491,"threads_started":2,"wal-append.queue_time_us":4701}
I20240503 14:15:07.141899 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(1bb01c0be7da4fe4b9c7c7160df7debc): 4838 bytes on disk
I20240503 14:15:07.142566 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(25eb1d6c445940d6bd8753a88381bb8c): perf score=0.033383
I20240503 14:15:07.143182 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(1bb01c0be7da4fe4b9c7c7160df7debc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 14:15:07.151162 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(f133ebe695a74f86be1a37ae661719ec): perf score=0.033381
I20240503 14:15:07.166612 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(25eb1d6c445940d6bd8753a88381bb8c): perf score=0.033382
I20240503 14:15:07.193506 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(1bb01c0be7da4fe4b9c7c7160df7debc) complete. Timing: real 0.080s user 0.039s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":68,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1414,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":930,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10474,"threads_started":1,"wal-append.queue_time_us":10574}
I20240503 14:15:07.197113 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(f133ebe695a74f86be1a37ae661719ec): 4923 bytes on disk
I20240503 14:15:07.197748 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(f133ebe695a74f86be1a37ae661719ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240503 14:15:07.203059 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(25eb1d6c445940d6bd8753a88381bb8c): perf score=0.033394
I20240503 14:15:07.229532 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(f133ebe695a74f86be1a37ae661719ec) complete. Timing: real 0.078s user 0.030s sys 0.029s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":287,"dirs.run_wall_time_us":1516,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":763,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14930,"threads_started":1,"wal-append.queue_time_us":15024}
I20240503 14:15:07.233590 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(f133ebe695a74f86be1a37ae661719ec): 4923 bytes on disk
I20240503 14:15:07.235414 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(f133ebe695a74f86be1a37ae661719ec) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240503 14:15:07.256377 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(25eb1d6c445940d6bd8753a88381bb8c) complete. Timing: real 0.113s user 0.043s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":23379,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":9528,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":746,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":23994,"threads_started":2,"wal-append.queue_time_us":554}
I20240503 14:15:07.259115 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(25eb1d6c445940d6bd8753a88381bb8c): 4583 bytes on disk
I20240503 14:15:07.260264 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(25eb1d6c445940d6bd8753a88381bb8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
I20240503 14:15:07.313961 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(25eb1d6c445940d6bd8753a88381bb8c) complete. Timing: real 0.147s user 0.056s sys 0.030s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31542,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":3812,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":927,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":38655,"threads_started":2,"wal-append.queue_time_us":7113}
I20240503 14:15:07.317585 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(25eb1d6c445940d6bd8753a88381bb8c): 4583 bytes on disk
I20240503 14:15:07.318315 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(25eb1d6c445940d6bd8753a88381bb8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240503 14:15:07.327697 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(25eb1d6c445940d6bd8753a88381bb8c) complete. Timing: real 0.123s user 0.047s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":4964,"drs_written":1,"lbm_read_time_us":118,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3867,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":231,"threads_started":1,"wal-append.queue_time_us":317}
I20240503 14:15:07.330428 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(1bb01c0be7da4fe4b9c7c7160df7debc): 4838 bytes on disk
I20240503 14:15:07.330968 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(1bb01c0be7da4fe4b9c7c7160df7debc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 14:15:07.332933 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(25eb1d6c445940d6bd8753a88381bb8c): 4583 bytes on disk
I20240503 14:15:07.333575 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(25eb1d6c445940d6bd8753a88381bb8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240503 14:15:07.599767 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:07.614401 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:07.641713 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:07.655436 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:07.698298 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:07.728859 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:07.740449 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:07.744769 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:07.911293 28432 consensus_queue.cc:237] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:07.913496 28304 raft_consensus.cc:1261] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:07.914361 31070 raft_consensus.cc:1261] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:07.914593 9252 consensus_queue.cc:1035] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:07.915390 9252 consensus_queue.cc:1035] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:07.925587 7344 raft_consensus.cc:2932] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:07.929018 31070 raft_consensus.cc:2932] T edd300244be74e3aa94d7ae35deeb470 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:07.931591 27899 catalog_manager.cc:5484] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:07.952904 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet edd300244be74e3aa94d7ae35deeb470 with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:15:07.953929 10035 tablet_replica.cc:330] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:15:07.954226 10035 raft_consensus.cc:2229] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:07.954548 10035 raft_consensus.cc:2258] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:07.955418 10035 ts_tablet_manager.cc:1875] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:07.956178 28304 raft_consensus.cc:2932] T edd300244be74e3aa94d7ae35deeb470 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:07.963106 10035 ts_tablet_manager.cc:1888] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:07.963264 10035 log.cc:1196] T edd300244be74e3aa94d7ae35deeb470 P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/edd300244be74e3aa94d7ae35deeb470
I20240503 14:15:07.964350 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet edd300244be74e3aa94d7ae35deeb470 (table rebalance_test_table_19 [id=00386c23ce26494485b6cc25f3ff5689]) successfully deleted
I20240503 14:15:08.326969 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:08.383241 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:08.449452 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:08.465498 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:08.472436 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:08.484285 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:08.528383 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:08.540253 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:08.726485 31201 consensus_queue.cc:237] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:08.728178 31072 raft_consensus.cc:1261] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:08.728284 28434 raft_consensus.cc:1261] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:08.728615 10108 consensus_queue.cc:1035] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:08.728955 9902 consensus_queue.cc:1035] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:08.731269 10108 raft_consensus.cc:2932] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:08.731870 31072 raft_consensus.cc:2932] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:08.732115 28434 raft_consensus.cc:2932] T bf316cde5631415faecbc1bb26690b79 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:08.741333 27899 catalog_manager.cc:5484] T bf316cde5631415faecbc1bb26690b79 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:08.749466 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet bf316cde5631415faecbc1bb26690b79 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:15:08.753225 10137 tablet_replica.cc:330] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:15:08.753475 10137 raft_consensus.cc:2229] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:08.753683 10137 raft_consensus.cc:2258] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:08.754346 10137 ts_tablet_manager.cc:1875] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:08.760190 10137 ts_tablet_manager.cc:1888] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:08.760386 10137 log.cc:1196] T bf316cde5631415faecbc1bb26690b79 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/bf316cde5631415faecbc1bb26690b79
I20240503 14:15:08.761130 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet bf316cde5631415faecbc1bb26690b79 (table rebalancer_extra_table_32 [id=6ad1408b592b40b48a38187cceb4c314]) successfully deleted
I20240503 14:15:09.091185 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:09.102610 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:09.108451 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:09.114897 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:09.136533 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:09.225589 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:09.243942 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:09.244088 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:09.354867 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(69d4d86cb68646bc9141c1e6df964561): perf score=0.033343
I20240503 14:15:09.361146 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(05a67063845044eea2d56c34cdfd4ab1): perf score=0.033352
I20240503 14:15:09.384435 28432 consensus_queue.cc:237] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:09.408182 31069 raft_consensus.cc:1261] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:09.409454 7344 consensus_queue.cc:1035] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:09.414503 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(69d4d86cb68646bc9141c1e6df964561) complete. Timing: real 0.059s user 0.053s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15317,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":861,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12910,"threads_started":2,"wal-append.queue_time_us":6388}
I20240503 14:15:09.415272 28304 raft_consensus.cc:1261] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:09.417114 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(69d4d86cb68646bc9141c1e6df964561): 4583 bytes on disk
I20240503 14:15:09.418130 7344 raft_consensus.cc:2932] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:09.421854 10240 consensus_queue.cc:1035] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:09.421976 31069 raft_consensus.cc:2932] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:09.426321 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(69d4d86cb68646bc9141c1e6df964561) complete. Timing: real 0.009s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":195,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.428807 27893 catalog_manager.cc:5484] T 213bc70b5cf14440ab88c754df0b8f7f P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:09.430591 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(05a67063845044eea2d56c34cdfd4ab1): perf score=0.033359
I20240503 14:15:09.449621 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(05a67063845044eea2d56c34cdfd4ab1) complete. Timing: real 0.088s user 0.042s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8110,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":17588,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":827,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":24249,"threads_started":2,"wal-append.queue_time_us":14942}
I20240503 14:15:09.452306 28154 tablet_service.cc:1514] Processing DeleteTablet for tablet 213bc70b5cf14440ab88c754df0b8f7f with delete_type TABLET_DATA_TOMBSTONED (TS 6409961ee8c0445687c11724213c44a9 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35238
I20240503 14:15:09.452965 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(69d4d86cb68646bc9141c1e6df964561): perf score=0.033376
I20240503 14:15:09.463563 28302 raft_consensus.cc:2932] T 213bc70b5cf14440ab88c754df0b8f7f P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:09.469467 10246 tablet_replica.cc:330] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: stopping tablet replica
I20240503 14:15:09.469772 10246 raft_consensus.cc:2229] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:09.470029 10246 raft_consensus.cc:2258] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:09.470746 10246 ts_tablet_manager.cc:1875] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:09.486455 10246 ts_tablet_manager.cc:1888] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:09.486682 10246 log.cc:1196] T 213bc70b5cf14440ab88c754df0b8f7f P 6409961ee8c0445687c11724213c44a9: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/wals/213bc70b5cf14440ab88c754df0b8f7f
I20240503 14:15:09.487742 27887 catalog_manager.cc:4830] TS 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): tablet 213bc70b5cf14440ab88c754df0b8f7f (table rebalance_test_table_13 [id=698fdaaae5954d1b8048ce09e5653803]) successfully deleted
I20240503 14:15:09.512953 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(05a67063845044eea2d56c34cdfd4ab1): perf score=0.033386
I20240503 14:15:09.515628 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(69d4d86cb68646bc9141c1e6df964561) complete. Timing: real 0.061s user 0.039s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":1326,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6446,"threads_started":1,"wal-append.queue_time_us":6354}
I20240503 14:15:09.517989 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1): 4923 bytes on disk
I20240503 14:15:09.519582 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1004,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.522205 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(529c250699cb4e09bae6129d138e37c2): perf score=0.033391
I20240503 14:15:09.534070 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(529c250699cb4e09bae6129d138e37c2): perf score=0.033390
I20240503 14:15:09.616974 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(529c250699cb4e09bae6129d138e37c2) complete. Timing: real 0.094s user 0.058s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":2397,"drs_written":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4,"lbm_write_time_us":912,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8540,"threads_started":1,"wal-append.queue_time_us":8450}
I20240503 14:15:09.619540 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2): 4838 bytes on disk
W20240503 14:15:09.619958 28107 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 49515.3us
I20240503 14:15:09.620136 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.622099 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(69d4d86cb68646bc9141c1e6df964561): 4583 bytes on disk
W20240503 14:15:09.627581 28109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 54885.6us
I20240503 14:15:09.637428 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(69d4d86cb68646bc9141c1e6df964561) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.674520 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(05a67063845044eea2d56c34cdfd4ab1) complete. Timing: real 0.244s user 0.122s sys 0.078s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":949,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":148641,"threads_started":1,"wal-append.queue_time_us":136206}
I20240503 14:15:09.677254 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1): 4923 bytes on disk
I20240503 14:15:09.677811 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.765637 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(529c250699cb4e09bae6129d138e37c2) complete. Timing: real 0.231s user 0.060s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7123,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":3198,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1039,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8652,"threads_started":2,"wal-append.queue_time_us":1715}
I20240503 14:15:09.768091 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2): 4838 bytes on disk
I20240503 14:15:09.768712 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.781410 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(05a67063845044eea2d56c34cdfd4ab1) complete. Timing: real 0.268s user 0.073s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8784,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":6697,"drs_written":1,"lbm_read_time_us":166,"lbm_reads_lt_1ms":4,"lbm_write_time_us":957,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":10955,"threads_started":2,"wal-append.queue_time_us":2128}
I20240503 14:15:09.783924 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(529c250699cb4e09bae6129d138e37c2): perf score=0.033457
I20240503 14:15:09.808393 28041 consensus_queue.cc:237] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:09.832020 30940 raft_consensus.cc:1261] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:09.832124 30807 raft_consensus.cc:1261] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:09.832784 27977 consensus_peers.cc:485] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f7bdba20d1844eefb6d3b29c5deb2873. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.832819 9031 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:09.833930 9608 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:09.839696 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(529c250699cb4e09bae6129d138e37c2) complete. Timing: real 0.055s user 0.040s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":952,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":291,"threads_started":1,"wal-append.queue_time_us":1518}
I20240503 14:15:09.842152 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1): 4923 bytes on disk
I20240503 14:15:09.842661 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.844633 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2): 4838 bytes on disk
I20240503 14:15:09.845126 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.846851 9031 raft_consensus.cc:2932] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:09.850080 30807 raft_consensus.cc:2932] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:09.853119 27899 catalog_manager.cc:5484] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:09.862479 30940 raft_consensus.cc:2932] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:09.917086 30940 consensus_queue.cc:237] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:09.919881 31200 raft_consensus.cc:1261] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:09.920496 9144 consensus_queue.cc:1035] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:09.920665 28041 raft_consensus.cc:1261] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:09.921797 30877 consensus_peers.cc:485] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5815f4c96eef4130aa0582d33a24b391. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.922960 9144 consensus_queue.cc:1035] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:09.942956 9799 raft_consensus.cc:2932] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:09.946653 31200 raft_consensus.cc:2932] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:09.948900 28041 raft_consensus.cc:2932] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:09.949290 27893 catalog_manager.cc:5484] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:09.985651 28171 consensus_queue.cc:237] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:09.988226 31072 raft_consensus.cc:1261] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:09.988732 28110 consensus_peers.cc:485] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f11b07607b0e430db646172f863cdc1f. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:09.990227 10256 consensus_queue.cc:1035] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:09.993430 30807 raft_consensus.cc:1261] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:09.993992 10256 consensus_queue.cc:1035] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:09.996584 10256 raft_consensus.cc:2932] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:09.998966 30807 raft_consensus.cc:2932] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:10.006592 27899 catalog_manager.cc:5484] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:10.008488 31072 raft_consensus.cc:2932] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:10.045830 28302 consensus_queue.cc:237] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:10.047981 28171 raft_consensus.cc:1261] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:15:10.048125 28237 consensus_peers.cc:485] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88f88e7eb7434450af76714256729887. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:10.048416 28040 raft_consensus.cc:1261] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:10.048556 9768 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:10.048961 10265 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:10.051326 9768 raft_consensus.cc:2932] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:10.051800 28171 raft_consensus.cc:2932] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:10.055532 28040 raft_consensus.cc:2932] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:10.059796 27893 catalog_manager.cc:5484] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 4, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:10.112123 28304 consensus_queue.cc:237] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:10.114316 30940 raft_consensus.cc:1261] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:10.114640 28238 consensus_peers.cc:485] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92f507cdcfb04eb4a9eb448773206783. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:10.114964 9768 consensus_queue.cc:1035] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:10.115268 31072 raft_consensus.cc:1261] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:10.116645 9768 consensus_queue.cc:1035] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:10.127432 9768 raft_consensus.cc:2932] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:10.127895 31072 raft_consensus.cc:2932] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:10.135890 27893 catalog_manager.cc:5484] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:10.136180 30940 raft_consensus.cc:2932] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:10.392382 10420 ts_tablet_manager.cc:902] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:10.419369 10423 ts_tablet_manager.cc:902] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:10.428829 10423 tablet_copy_client.cc:308] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:15:10.429412 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet f7bdba20d1844eefb6d3b29c5deb2873 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:41995)
I20240503 14:15:10.429543 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet f7bdba20d1844eefb6d3b29c5deb2873 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:41995: session id = 6409961ee8c0445687c11724213c44a9-f7bdba20d1844eefb6d3b29c5deb2873
I20240503 14:15:10.431732 28063 tablet_copy_source_session.cc:205] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:10.434803 10423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7bdba20d1844eefb6d3b29c5deb2873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.437471 10420 tablet_copy_client.cc:308] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:15:10.441406 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 5815f4c96eef4130aa0582d33a24b391 from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:36635)
I20240503 14:15:10.441540 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 5815f4c96eef4130aa0582d33a24b391 from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:36635: session id = 5388902973134958a030fd63ab4faba4-5815f4c96eef4130aa0582d33a24b391
I20240503 14:15:10.444187 30961 tablet_copy_source_session.cc:205] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:10.444232 10423 tablet_copy_client.cc:783] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.444520 10423 tablet_copy_client.cc:647] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.447590 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5815f4c96eef4130aa0582d33a24b391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.449304 10423 tablet_copy_client.cc:523] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.454777 10423 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:15:10.467499 10420 tablet_copy_client.cc:783] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.467872 10420 tablet_copy_client.cc:647] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.469406 10420 tablet_copy_client.cc:523] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.483299 10420 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:15:10.485073 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:10.485174 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:10.499481 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:10.499775 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:10.538152 10420 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.538807 10420 tablet_bootstrap.cc:492] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:15:10.539208 10420 ts_tablet_manager.cc:1367] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.001s
I20240503 14:15:10.542474 10420 raft_consensus.cc:348] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:10.546437 10420 raft_consensus.cc:729] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:15:10.546792 10420 consensus_queue.cc:260] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:10.549961 10420 ts_tablet_manager.cc:1398] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 14:15:10.556736 10423 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.557241 10423 tablet_bootstrap.cc:492] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:15:10.557554 10423 ts_tablet_manager.cc:1367] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.006s
I20240503 14:15:10.558487 10423 raft_consensus.cc:348] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:10.558794 10423 raft_consensus.cc:729] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:15:10.559041 10423 consensus_queue.cc:260] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:10.559644 10423 ts_tablet_manager.cc:1398] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:10.562343 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-5815f4c96eef4130aa0582d33a24b391 received from {username='slave'} at 127.20.240.68:36635
I20240503 14:15:10.562481 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 5388902973134958a030fd63ab4faba4-5815f4c96eef4130aa0582d33a24b391 on tablet 5815f4c96eef4130aa0582d33a24b391 with peer 5388902973134958a030fd63ab4faba4
I20240503 14:15:10.608836 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:10.610720 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-f7bdba20d1844eefb6d3b29c5deb2873 received from {username='slave'} at 127.20.240.66:41995
I20240503 14:15:10.610849 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 6409961ee8c0445687c11724213c44a9-f7bdba20d1844eefb6d3b29c5deb2873 on tablet f7bdba20d1844eefb6d3b29c5deb2873 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:15:10.624645 10441 ts_tablet_manager.cc:902] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:10.648448 10447 ts_tablet_manager.cc:902] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:10.666270 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:10.667724 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:10.725179 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:10.736676 10423 ts_tablet_manager.cc:902] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:10.753757 10447 tablet_copy_client.cc:308] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:15:10.754431 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet f11b07607b0e430db646172f863cdc1f from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:50673)
I20240503 14:15:10.754586 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet f11b07607b0e430db646172f863cdc1f from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:50673: session id = 793a0d5dd1634c5faaf0e1e4851e8294-f11b07607b0e430db646172f863cdc1f
I20240503 14:15:10.757576 28194 tablet_copy_source_session.cc:205] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:10.758734 10447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11b07607b0e430db646172f863cdc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.764925 10447 tablet_copy_client.cc:783] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.765236 10447 tablet_copy_client.cc:647] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.766736 10447 tablet_copy_client.cc:523] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.770221 10447 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:15:10.796267 10423 tablet_copy_client.cc:308] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:15:10.797106 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 92f507cdcfb04eb4a9eb448773206783 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:56239)
I20240503 14:15:10.797317 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 92f507cdcfb04eb4a9eb448773206783 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:56239: session id = 6409961ee8c0445687c11724213c44a9-92f507cdcfb04eb4a9eb448773206783
I20240503 14:15:10.807101 28324 tablet_copy_source_session.cc:205] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:10.808840 10423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92f507cdcfb04eb4a9eb448773206783. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.816722 10423 tablet_copy_client.cc:783] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:10.816953 10423 tablet_copy_client.cc:647] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.821272 10441 tablet_copy_client.cc:308] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.67:43889
I20240503 14:15:10.821848 28324 tablet_copy_service.cc:140] P a4cdf05256a0407c8929ac43eb55d44f: Received BeginTabletCopySession request for tablet 88f88e7eb7434450af76714256729887 from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:53837)
I20240503 14:15:10.821966 28324 tablet_copy_service.cc:161] P a4cdf05256a0407c8929ac43eb55d44f: Beginning new tablet copy session on tablet 88f88e7eb7434450af76714256729887 from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:53837: session id = d0dbfb0e43f948c3922926aacc365fc1-88f88e7eb7434450af76714256729887
I20240503 14:15:10.822002 10423 tablet_copy_client.cc:523] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.825413 10423 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:15:10.828557 28324 tablet_copy_source_session.cc:205] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:15:10.828889 10447 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.829345 10447 tablet_bootstrap.cc:492] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:15:10.830791 10447 ts_tablet_manager.cc:1367] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.030s
I20240503 14:15:10.831408 10447 raft_consensus.cc:348] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:10.831584 10447 raft_consensus.cc:729] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:15:10.834273 10447 consensus_queue.cc:260] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:10.835336 10441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f88e7eb7434450af76714256729887. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:10.837276 10447 ts_tablet_manager.cc:1398] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:15:10.841812 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-f11b07607b0e430db646172f863cdc1f received from {username='slave'} at 127.20.240.65:50673
I20240503 14:15:10.842458 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-f11b07607b0e430db646172f863cdc1f on tablet f11b07607b0e430db646172f863cdc1f with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:10.847018 10441 tablet_copy_client.cc:783] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 5 data blocks...
I20240503 14:15:10.863844 10423 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.864372 10423 tablet_bootstrap.cc:492] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:15:10.864656 10423 ts_tablet_manager.cc:1367] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.026s
I20240503 14:15:10.865415 10423 raft_consensus.cc:348] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:10.865628 10441 tablet_copy_client.cc:647] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:10.865674 10423 raft_consensus.cc:729] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:15:10.865942 10423 consensus_queue.cc:260] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:10.866830 10423 ts_tablet_manager.cc:1398] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:10.867662 28323 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-92f507cdcfb04eb4a9eb448773206783 received from {username='slave'} at 127.20.240.66:56239
I20240503 14:15:10.867756 28323 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session 6409961ee8c0445687c11724213c44a9-92f507cdcfb04eb4a9eb448773206783 on tablet 92f507cdcfb04eb4a9eb448773206783 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:15:10.872355 10441 tablet_copy_client.cc:523] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:10.876461 10441 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:15:10.916971 28171 raft_consensus.cc:1203] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:10.917507 10441 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:10.917959 10441 tablet_bootstrap.cc:492] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
I20240503 14:15:10.918285 10441 ts_tablet_manager.cc:1367] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.042s user 0.020s sys 0.003s
I20240503 14:15:10.918958 10441 raft_consensus.cc:348] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:10.919286 10441 raft_consensus.cc:729] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:15:10.919572 10441 consensus_queue.cc:260] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:10.920267 10441 ts_tablet_manager.cc:1398] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:10.921018 28324 tablet_copy_service.cc:342] P a4cdf05256a0407c8929ac43eb55d44f: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-88f88e7eb7434450af76714256729887 received from {username='slave'} at 127.20.240.69:53837
I20240503 14:15:10.921113 28324 tablet_copy_service.cc:433] P a4cdf05256a0407c8929ac43eb55d44f: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-88f88e7eb7434450af76714256729887 on tablet 88f88e7eb7434450af76714256729887 with peer d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:15:10.924983 30856 maintenance_manager.cc:419] P d0dbfb0e43f948c3922926aacc365fc1: Scheduling UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887): 4924 bytes on disk
I20240503 14:15:10.926379 30747 maintenance_manager.cc:643] P d0dbfb0e43f948c3922926aacc365fc1: UndoDeltaBlockGCOp(88f88e7eb7434450af76714256729887) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240503 14:15:10.969482 30810 raft_consensus.cc:1203] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:10.979254 28434 raft_consensus.cc:1203] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:11.152257 28041 raft_consensus.cc:1203] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:11.284801 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:11.299872 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:11.395023 28171 raft_consensus.cc:1203] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:11.411935 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:11.534931 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240503 14:15:11.571525 31005 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 66614.6us
I20240503 14:15:11.579732 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:11.580140 28283 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:11.602043 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:11.610230 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:11.730134 28302 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88f88e7eb7434450af76714256729887"
dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40280
I20240503 14:15:11.730381 28302 raft_consensus.cc:595] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:11.995927 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:12.011010 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:12.016894 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:12.021406 9768 raft_consensus.cc:979] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: : Instructing follower 793a0d5dd1634c5faaf0e1e4851e8294 to start an election
I20240503 14:15:12.021680 9768 raft_consensus.cc:1067] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Signalling peer 793a0d5dd1634c5faaf0e1e4851e8294 to start an election
I20240503 14:15:12.022523 28043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88f88e7eb7434450af76714256729887"
dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
from {username='slave'} at 127.20.240.67:53677
I20240503 14:15:12.022734 28043 raft_consensus.cc:482] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:12.022856 28043 raft_consensus.cc:3037] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:12.025401 28043 raft_consensus.cc:504] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:12.026770 28043 leader_election.cc:288] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:12.027493 28171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f88e7eb7434450af76714256729887" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:15:12.027719 28171 raft_consensus.cc:3037] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:12.031664 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f88e7eb7434450af76714256729887" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:15:12.031925 30810 raft_consensus.cc:3037] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:12.035091 30810 raft_consensus.cc:2454] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:12.037418 28302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88f88e7eb7434450af76714256729887" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:15:12.037606 28302 raft_consensus.cc:3032] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:12.037679 28302 raft_consensus.cc:729] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Running, Role: LEADER
I20240503 14:15:12.038012 28302 consensus_queue.cc:260] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:12.038513 28302 raft_consensus.cc:3037] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:12.041956 28302 raft_consensus.cc:2454] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:12.042358 27979 leader_election.cc:302] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:15:12.042657 9031 raft_consensus.cc:2781] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:12.042804 9031 raft_consensus.cc:686] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:12.043074 9031 consensus_queue.cc:237] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:12.048089 28171 raft_consensus.cc:2454] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:12.050024 27902 catalog_manager.cc:5484] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:12.068719 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:12.076442 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:12.080308 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:12.126206 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:12.170697 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:12.459945 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(859456c340dc47bf8fc600ebe8c4b984): perf score=0.033334
I20240503 14:15:12.556063 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(859456c340dc47bf8fc600ebe8c4b984): perf score=0.033359
I20240503 14:15:12.625998 28302 raft_consensus.cc:1261] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:15:12.627059 30808 raft_consensus.cc:1261] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:15:12.627646 10287 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:12.628007 9031 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:12.639369 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(30e4fb0a79814959aa25cf6eb34dfbb9): perf score=0.033383
I20240503 14:15:12.644834 28171 raft_consensus.cc:1261] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:15:12.646477 9031 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:12.676100 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(8868e8d0fd6b4a14970df92960ca0b80): perf score=0.033392
W20240503 14:15:13.062824 30877 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.65:33293, user_credentials={real_user=slave}} blocked reactor thread for 184650us
I20240503 14:15:13.111379 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(859456c340dc47bf8fc600ebe8c4b984) complete. Timing: real 0.651s user 0.054s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":28852,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":1614,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6674,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":40005,"threads_started":2,"wal-append.queue_time_us":11052}
I20240503 14:15:13.116850 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(859456c340dc47bf8fc600ebe8c4b984): 4583 bytes on disk
I20240503 14:15:13.117516 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(859456c340dc47bf8fc600ebe8c4b984) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
W20240503 14:15:13.147310 28111 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.422s user 0.128s sys 0.262s
I20240503 14:15:13.156951 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(30e4fb0a79814959aa25cf6eb34dfbb9) complete. Timing: real 0.517s user 0.163s sys 0.268s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":26431,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":4979,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":447259,"threads_started":2,"wal-append.queue_time_us":421430}
I20240503 14:15:13.161211 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(30e4fb0a79814959aa25cf6eb34dfbb9): 4838 bytes on disk
I20240503 14:15:13.162114 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(30e4fb0a79814959aa25cf6eb34dfbb9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":195,"lbm_reads_lt_1ms":4}
I20240503 14:15:13.259599 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:13.280937 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:13.291606 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(8868e8d0fd6b4a14970df92960ca0b80) complete. Timing: real 0.615s user 0.079s sys 0.103s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":430328,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":17588,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1078,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":466026,"threads_started":2,"wal-append.queue_time_us":22805}
I20240503 14:15:13.304472 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(8868e8d0fd6b4a14970df92960ca0b80): 4923 bytes on disk
I20240503 14:15:13.306841 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(8868e8d0fd6b4a14970df92960ca0b80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
I20240503 14:15:13.310170 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(30e4fb0a79814959aa25cf6eb34dfbb9): perf score=0.033565
I20240503 14:15:13.319593 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(859456c340dc47bf8fc600ebe8c4b984) complete. Timing: real 0.763s user 0.046s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8350,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":7820,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":33100,"threads_started":2,"wal-append.queue_time_us":12036}
I20240503 14:15:13.335683 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(859456c340dc47bf8fc600ebe8c4b984): 4583 bytes on disk
I20240503 14:15:13.335935 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:13.338692 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(859456c340dc47bf8fc600ebe8c4b984) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 14:15:13.345317 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(8868e8d0fd6b4a14970df92960ca0b80): perf score=0.033577
I20240503 14:15:13.403445 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:13.432188 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:13.444007 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:13.457015 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:13.501840 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:13.544212 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(8868e8d0fd6b4a14970df92960ca0b80) complete. Timing: real 0.193s user 0.047s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":72,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":1812,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":818,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12302,"threads_started":1,"wal-append.queue_time_us":12398}
I20240503 14:15:13.547029 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(8868e8d0fd6b4a14970df92960ca0b80): 4923 bytes on disk
I20240503 14:15:13.548919 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(8868e8d0fd6b4a14970df92960ca0b80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
I20240503 14:15:13.602629 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(30e4fb0a79814959aa25cf6eb34dfbb9) complete. Timing: real 0.280s user 0.047s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1032,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1831,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":21743,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":17309,"threads_started":1,"wal-append.queue_time_us":17416}
I20240503 14:15:13.605746 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(30e4fb0a79814959aa25cf6eb34dfbb9): 4838 bytes on disk
I20240503 14:15:13.606487 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(30e4fb0a79814959aa25cf6eb34dfbb9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":245,"lbm_reads_lt_1ms":4}
I20240503 14:15:13.713557 28171 consensus_queue.cc:237] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:13.715286 28040 raft_consensus.cc:1261] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:13.715273 31070 raft_consensus.cc:1261] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:13.715801 9713 consensus_queue.cc:1035] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:13.716042 10256 consensus_queue.cc:1035] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:13.718438 10256 raft_consensus.cc:2932] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:13.726527 28040 raft_consensus.cc:2932] T f11b07607b0e430db646172f863cdc1f P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:13.732192 31070 raft_consensus.cc:2932] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:13.738862 27894 catalog_manager.cc:5484] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:13.748378 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet f11b07607b0e430db646172f863cdc1f with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:15:13.749195 10731 tablet_replica.cc:330] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:15:13.749414 10731 raft_consensus.cc:2229] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:13.749686 10731 raft_consensus.cc:2258] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:13.750574 10731 ts_tablet_manager.cc:1875] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:13.757086 10731 ts_tablet_manager.cc:1888] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:13.757202 10731 log.cc:1196] T f11b07607b0e430db646172f863cdc1f P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/f11b07607b0e430db646172f863cdc1f
I20240503 14:15:13.757833 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet f11b07607b0e430db646172f863cdc1f (table rebalancer_extra_table_40 [id=4704a39445e44ee2b1d1b44d499d33df]) successfully deleted
I20240503 14:15:14.087082 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:14.089097 28154 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:14.092072 28021 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:14.094192 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:14.127945 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:14.130887 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:14.133975 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:14.184652 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:14.321172 28043 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873"
dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58210
I20240503 14:15:14.321368 28043 raft_consensus.cc:595] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:14.476087 10662 raft_consensus.cc:979] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: : Instructing follower 6409961ee8c0445687c11724213c44a9 to start an election
I20240503 14:15:14.476218 10662 raft_consensus.cc:1067] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Signalling peer 6409961ee8c0445687c11724213c44a9 to start an election
I20240503 14:15:14.477806 28173 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873"
dest_uuid: "6409961ee8c0445687c11724213c44a9"
from {username='slave'} at 127.20.240.65:50673
I20240503 14:15:14.478000 28173 raft_consensus.cc:482] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:14.478109 28173 raft_consensus.cc:3037] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.480278 28173 raft_consensus.cc:504] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:14.481231 28173 leader_election.cc:288] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:14.481520 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:15:14.481701 28040 raft_consensus.cc:3032] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:14.481649 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:15:14.481762 28040 raft_consensus.cc:729] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:14.481863 30810 raft_consensus.cc:3037] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.481981 28040 consensus_queue.cc:260] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:14.483983 30941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7bdba20d1844eefb6d3b29c5deb2873" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
I20240503 14:15:14.484164 30941 raft_consensus.cc:3037] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.484608 30810 raft_consensus.cc:2454] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:15:14.486996 30941 raft_consensus.cc:2454] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:15:14.488152 28109 leader_election.cc:302] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:15:14.488478 10726 raft_consensus.cc:2781] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:14.488631 28040 raft_consensus.cc:3037] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:14.488698 10726 raft_consensus.cc:686] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:15:14.489256 10726 consensus_queue.cc:237] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:14.495404 28040 raft_consensus.cc:2454] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:15:14.495529 27894 catalog_manager.cc:5484] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 1 to 2, leader changed from 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:14.680440 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(dea30267166942258d47155fd0173713): perf score=0.033336
I20240503 14:15:14.811318 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:14.818606 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(5f25a3bcdef849499e4a02f01f4dbc0e): perf score=0.033365
I20240503 14:15:14.853559 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:14.869585 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:14.874826 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:14.882915 30808 raft_consensus.cc:1261] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:14.883507 10868 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:14.886432 28043 raft_consensus.cc:1261] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:14.887544 10868 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:14.889035 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(213bc70b5cf14440ab88c754df0b8f7f): perf score=0.033386
I20240503 14:15:14.900099 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(5f25a3bcdef849499e4a02f01f4dbc0e): perf score=0.033386
I20240503 14:15:14.931177 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:14.965349 30939 raft_consensus.cc:1261] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:14.965961 10726 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:14.997689 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:14.999500 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:15.037421 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(5f25a3bcdef849499e4a02f01f4dbc0e) complete. Timing: real 0.219s user 0.042s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":22009,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1585,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":836,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":55148,"threads_started":2,"wal-append.queue_time_us":36155}
I20240503 14:15:15.052052 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(5f25a3bcdef849499e4a02f01f4dbc0e): 4838 bytes on disk
I20240503 14:15:15.056461 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:15.060405 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(5f25a3bcdef849499e4a02f01f4dbc0e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 14:15:15.076932 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(5f25a3bcdef849499e4a02f01f4dbc0e) complete. Timing: real 0.177s user 0.058s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":9955,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":13119,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":876,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16447,"threads_started":2,"wal-append.queue_time_us":6279}
I20240503 14:15:15.079806 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(5f25a3bcdef849499e4a02f01f4dbc0e): 4838 bytes on disk
I20240503 14:15:15.087924 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(5f25a3bcdef849499e4a02f01f4dbc0e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 14:15:15.093303 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(213bc70b5cf14440ab88c754df0b8f7f): perf score=0.033437
I20240503 14:15:15.192202 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(dea30267166942258d47155fd0173713) complete. Timing: real 0.511s user 0.030s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":79887,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":6095,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5142,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":182247,"threads_started":2,"wal-append.queue_time_us":47148}
I20240503 14:15:15.194885 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(dea30267166942258d47155fd0173713): 4583 bytes on disk
I20240503 14:15:15.196440 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(dea30267166942258d47155fd0173713) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240503 14:15:15.198840 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(213bc70b5cf14440ab88c754df0b8f7f) complete. Timing: real 0.305s user 0.036s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":310,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":8473,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5413,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15603,"threads_started":2,"wal-append.queue_time_us":11747}
I20240503 14:15:15.205935 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(dea30267166942258d47155fd0173713): perf score=0.033470
I20240503 14:15:15.207633 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(213bc70b5cf14440ab88c754df0b8f7f) complete. Timing: real 0.114s user 0.060s sys 0.034s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":207,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1577,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1004,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":49821,"threads_started":1,"wal-append.queue_time_us":49906}
I20240503 14:15:15.210719 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(213bc70b5cf14440ab88c754df0b8f7f): 4923 bytes on disk
I20240503 14:15:15.211341 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(213bc70b5cf14440ab88c754df0b8f7f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 14:15:15.268842 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(dea30267166942258d47155fd0173713) complete. Timing: real 0.063s user 0.039s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1380,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6741,"threads_started":1,"wal-append.queue_time_us":6640}
I20240503 14:15:15.272415 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(213bc70b5cf14440ab88c754df0b8f7f): 4923 bytes on disk
I20240503 14:15:15.273008 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(213bc70b5cf14440ab88c754df0b8f7f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 14:15:15.275523 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(dea30267166942258d47155fd0173713): 4583 bytes on disk
I20240503 14:15:15.276399 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(dea30267166942258d47155fd0173713) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 14:15:15.639045 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:15.663600 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:15.689718 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:15.697634 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:15.703387 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:15.731410 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:15.757221 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 14:15:15.774318 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 93939.3us
I20240503 14:15:15.818209 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:15.961925 28043 consensus_queue.cc:237] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:15.964112 31200 raft_consensus.cc:1261] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:15.964085 28433 raft_consensus.cc:1261] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:15.964576 10662 consensus_queue.cc:1035] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:15.964855 10661 consensus_queue.cc:1035] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:15.967437 10662 raft_consensus.cc:2932] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:15.969128 28433 raft_consensus.cc:2932] T 25eb1d6c445940d6bd8753a88381bb8c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:15.970950 31200 raft_consensus.cc:2932] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:15.975368 27898 catalog_manager.cc:5484] T 25eb1d6c445940d6bd8753a88381bb8c P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:15.995579 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 25eb1d6c445940d6bd8753a88381bb8c with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43626
I20240503 14:15:15.996119 10993 tablet_replica.cc:330] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:15:15.996358 10993 raft_consensus.cc:2229] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:15.996551 10993 raft_consensus.cc:2258] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:15.997468 10993 ts_tablet_manager.cc:1875] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:16.011015 10993 ts_tablet_manager.cc:1888] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:16.011123 10993 log.cc:1196] T 25eb1d6c445940d6bd8753a88381bb8c P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/25eb1d6c445940d6bd8753a88381bb8c
I20240503 14:15:16.011837 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 25eb1d6c445940d6bd8753a88381bb8c (table rebalance_test_table_7 [id=914cf0996edd4e549f6daa8a3210a29d]) successfully deleted
I20240503 14:15:16.357318 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:16.380563 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:16.389045 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:16.404904 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:16.428720 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:16.447628 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:16.459784 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:16.567628 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:16.726332 30941 consensus_queue.cc:237] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:16.730394 28433 raft_consensus.cc:1261] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:16.731104 11047 consensus_queue.cc:1035] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:16.731331 28040 raft_consensus.cc:1261] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:16.732136 11047 consensus_queue.cc:1035] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:16.733979 11047 raft_consensus.cc:2932] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.739428 28433 raft_consensus.cc:2932] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.740427 28040 raft_consensus.cc:2932] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.741237 27896 catalog_manager.cc:5484] T 5815f4c96eef4130aa0582d33a24b391 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.748890 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 5815f4c96eef4130aa0582d33a24b391 with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:15:16.749413 11087 tablet_replica.cc:330] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:15:16.749606 11087 raft_consensus.cc:2229] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:16.749781 11087 raft_consensus.cc:2258] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:16.750371 11087 ts_tablet_manager.cc:1875] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:16.756471 11087 ts_tablet_manager.cc:1888] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:16.756651 11087 log.cc:1196] T 5815f4c96eef4130aa0582d33a24b391 P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/5815f4c96eef4130aa0582d33a24b391
I20240503 14:15:16.757390 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 5815f4c96eef4130aa0582d33a24b391 (table rebalancer_extra_table_14 [id=bc773fb7d5424a3dacb44ca3271ac4ae]) successfully deleted
I20240503 14:15:16.898124 31070 consensus_queue.cc:237] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:16.902469 28171 raft_consensus.cc:1261] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:16.902606 31200 raft_consensus.cc:1261] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:16.903347 11063 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:16.903928 11063 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:16.904276 31008 consensus_peers.cc:485] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1af2fb3fb3fd4dfe873d124232d812b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:16.918391 11063 raft_consensus.cc:2932] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.924715 28173 raft_consensus.cc:2932] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.940711 31200 raft_consensus.cc:2932] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.937201 27899 catalog_manager.cc:5484] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:16.991441 30807 consensus_queue.cc:237] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:17.002600 28302 raft_consensus.cc:1261] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:17.002614 30744 consensus_peers.cc:485] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6151ea340334090a5c0e79ca65af5c1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.003038 30941 raft_consensus.cc:1261] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.003211 10539 consensus_queue.cc:1035] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.003582 10792 consensus_queue.cc:1035] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.007014 10792 raft_consensus.cc:2932] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.007680 28302 raft_consensus.cc:2932] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.011184 30941 raft_consensus.cc:2932] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.016630 27893 catalog_manager.cc:5484] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.066640 30807 consensus_queue.cc:237] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:17.069406 31070 raft_consensus.cc:1261] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:17.069463 30744 consensus_peers.cc:485] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d4c447b816b430caf4b213f896e51e3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.069532 30941 raft_consensus.cc:1261] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.071215 10539 consensus_queue.cc:1035] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.071780 11181 consensus_queue.cc:1035] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.084949 10539 raft_consensus.cc:2932] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.089831 30941 raft_consensus.cc:2932] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.089684 31070 raft_consensus.cc:2932] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.096971 27896 catalog_manager.cc:5484] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.150053 30941 consensus_queue.cc:237] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:17.152273 30807 raft_consensus.cc:1261] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.152318 31200 raft_consensus.cc:1261] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:17.152482 30877 consensus_peers.cc:485] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72c7ff14d43342439f935389b03ecad0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.152858 10962 consensus_queue.cc:1035] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.153084 10463 consensus_queue.cc:1035] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.167704 10962 raft_consensus.cc:2932] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:17.168334 30807 raft_consensus.cc:2932] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:17.173012 31200 raft_consensus.cc:2932] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:17.180474 27893 catalog_manager.cc:5484] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:17.219777 28040 consensus_queue.cc:237] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:17.222272 28302 raft_consensus.cc:1261] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:17.222702 28173 raft_consensus.cc:1261] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:17.223145 10983 consensus_queue.cc:1035] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:17.223374 27977 consensus_peers.cc:485] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 529c250699cb4e09bae6129d138e37c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.223664 10983 consensus_queue.cc:1035] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:17.227952 11236 raft_consensus.cc:2932] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.228896 28173 raft_consensus.cc:2932] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.232024 28302 raft_consensus.cc:2932] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.234510 27899 catalog_manager.cc:5484] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 4, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:17.298522 28433 consensus_queue.cc:237] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:17.301170 28173 raft_consensus.cc:1261] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:15:17.301246 28302 raft_consensus.cc:1261] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:15:17.301471 28369 consensus_peers.cc:485] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05a67063845044eea2d56c34cdfd4ab1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.301802 11256 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:17.302067 10784 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:17.315294 10784 raft_consensus.cc:2932] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:17.315853 28302 raft_consensus.cc:2932] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:17.320565 28173 raft_consensus.cc:2932] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:17.322322 27893 catalog_manager.cc:5484] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index -1 to 4, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:17.362272 28043 consensus_queue.cc:237] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:17.364375 30807 raft_consensus.cc:1261] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.364753 31070 raft_consensus.cc:1261] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.364939 10983 consensus_queue.cc:1035] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:17.365548 27977 consensus_peers.cc:485] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbd2cac7ae8e45b6bd407b119efe3a51. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.365832 10983 consensus_queue.cc:1035] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.379431 10662 raft_consensus.cc:2932] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.379954 31069 raft_consensus.cc:2932] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.385597 30807 raft_consensus.cc:2932] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.389642 27896 catalog_manager.cc:5484] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.452934 28043 consensus_queue.cc:237] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
W20240503 14:15:17.455235 27979 consensus_peers.cc:485] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06e418c4c2cf46608ba454d48fae9285. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.455281 31200 raft_consensus.cc:1261] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.455307 30807 raft_consensus.cc:1261] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.455775 10983 consensus_queue.cc:1035] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.456127 10662 consensus_queue.cc:1035] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.459249 10983 raft_consensus.cc:2932] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:17.459743 30807 raft_consensus.cc:2932] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:17.464143 31200 raft_consensus.cc:2932] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:17.466269 27893 catalog_manager.cc:5484] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:17.501639 28434 consensus_queue.cc:237] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:15:17.518884 28368 consensus_peers.cc:485] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 130a2a0b5aa344d3be535cfde8990d39. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.520002 28043 raft_consensus.cc:1261] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.520772 10784 consensus_queue.cc:1035] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.529330 30807 raft_consensus.cc:1261] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.530195 11315 ts_tablet_manager.cc:902] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:17.530134 10784 consensus_queue.cc:1035] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.534428 10784 raft_consensus.cc:2932] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.537266 11313 ts_tablet_manager.cc:902] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:17.539587 11312 ts_tablet_manager.cc:902] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:17.539440 30807 raft_consensus.cc:2932] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.548308 28040 raft_consensus.cc:2932] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.548497 27893 catalog_manager.cc:5484] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:17.560372 11312 tablet_copy_client.cc:308] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:15:17.561301 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet f6151ea340334090a5c0e79ca65af5c1 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:50623)
I20240503 14:15:17.561475 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet f6151ea340334090a5c0e79ca65af5c1 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:50623: session id = 530893a961af4b95b6b687fa5ed22f9d-f6151ea340334090a5c0e79ca65af5c1
I20240503 14:15:17.565393 30830 tablet_copy_source_session.cc:205] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:17.567960 11312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6151ea340334090a5c0e79ca65af5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:17.569528 11313 tablet_copy_client.cc:308] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:15:17.570976 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet 1af2fb3fb3fd4dfe873d124232d812b5 from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:50449)
I20240503 14:15:17.571111 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet 1af2fb3fb3fd4dfe873d124232d812b5 from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:50449: session id = 5388902973134958a030fd63ab4faba4-1af2fb3fb3fd4dfe873d124232d812b5
I20240503 14:15:17.574169 31092 tablet_copy_source_session.cc:205] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:17.574836 11312 tablet_copy_client.cc:783] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:17.575107 11312 tablet_copy_client.cc:647] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:17.576432 11313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1af2fb3fb3fd4dfe873d124232d812b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:17.578061 11312 tablet_copy_client.cc:523] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:17.583706 11313 tablet_copy_client.cc:783] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:17.583804 11312 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:15:17.584002 11313 tablet_copy_client.cc:647] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:17.585737 11313 tablet_copy_client.cc:523] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:17.592154 11313 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:15:17.599558 11313 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:17.599892 11313 tablet_bootstrap.cc:492] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:15:17.600140 11313 ts_tablet_manager.cc:1367] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240503 14:15:17.600765 11313 raft_consensus.cc:348] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:17.600956 11313 raft_consensus.cc:729] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:15:17.601150 11313 consensus_queue.cc:260] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:17.602349 11313 ts_tablet_manager.cc:1398] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:15:17.603178 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-1af2fb3fb3fd4dfe873d124232d812b5 received from {username='slave'} at 127.20.240.68:50449
I20240503 14:15:17.603281 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session 5388902973134958a030fd63ab4faba4-1af2fb3fb3fd4dfe873d124232d812b5 on tablet 1af2fb3fb3fd4dfe873d124232d812b5 with peer 5388902973134958a030fd63ab4faba4
I20240503 14:15:17.604821 11315 tablet_copy_client.cc:308] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:15:17.606034 11312 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:17.606142 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 72c7ff14d43342439f935389b03ecad0 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:45069)
I20240503 14:15:17.606302 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 72c7ff14d43342439f935389b03ecad0 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:45069: session id = 793a0d5dd1634c5faaf0e1e4851e8294-72c7ff14d43342439f935389b03ecad0
I20240503 14:15:17.606400 11312 tablet_bootstrap.cc:492] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:15:17.606616 11312 ts_tablet_manager.cc:1367] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.002s
I20240503 14:15:17.607193 11312 raft_consensus.cc:348] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:17.607379 11312 raft_consensus.cc:729] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:15:17.607558 11312 consensus_queue.cc:260] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:17.608017 11312 ts_tablet_manager.cc:1398] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:17.608692 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-f6151ea340334090a5c0e79ca65af5c1 received from {username='slave'} at 127.20.240.72:50623
I20240503 14:15:17.608783 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-f6151ea340334090a5c0e79ca65af5c1 on tablet f6151ea340334090a5c0e79ca65af5c1 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:15:17.608965 30961 tablet_copy_source_session.cc:205] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:17.610462 11315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72c7ff14d43342439f935389b03ecad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:17.618081 11315 tablet_copy_client.cc:783] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:17.618345 11315 tablet_copy_client.cc:647] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:17.619904 11315 tablet_copy_client.cc:523] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:17.620633 11337 ts_tablet_manager.cc:902] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:17.629127 11315 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:15:17.642294 11315 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:17.642812 11315 tablet_bootstrap.cc:492] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:15:17.643151 11315 ts_tablet_manager.cc:1367] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240503 14:15:17.644289 11315 raft_consensus.cc:348] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:17.644675 11315 raft_consensus.cc:729] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:15:17.645665 11315 consensus_queue.cc:260] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:17.646497 11315 ts_tablet_manager.cc:1398] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:17.647301 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-72c7ff14d43342439f935389b03ecad0 received from {username='slave'} at 127.20.240.65:45069
I20240503 14:15:17.647408 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-72c7ff14d43342439f935389b03ecad0 on tablet 72c7ff14d43342439f935389b03ecad0 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:17.654381 11337 tablet_copy_client.cc:308] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:15:17.655042 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet 4d4c447b816b430caf4b213f896e51e3 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:46535)
I20240503 14:15:17.655177 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet 4d4c447b816b430caf4b213f896e51e3 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:46535: session id = 6409961ee8c0445687c11724213c44a9-4d4c447b816b430caf4b213f896e51e3
I20240503 14:15:17.658087 30830 tablet_copy_source_session.cc:205] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:17.677515 11337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d4c447b816b430caf4b213f896e51e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:17.687454 11337 tablet_copy_client.cc:783] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:17.687705 11337 tablet_copy_client.cc:647] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:17.690135 11337 tablet_copy_client.cc:523] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:17.691313 11312 ts_tablet_manager.cc:902] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:17.695672 28043 consensus_queue.cc:237] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:17.697238 11337 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
W20240503 14:15:17.701205 27979 consensus_peers.cc:485] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff8d8c22d99c462b931a77297c5ff1ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.702036 30941 raft_consensus.cc:1261] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.702487 31200 raft_consensus.cc:1261] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.702683 10983 consensus_queue.cc:1035] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.704675 10662 consensus_queue.cc:1035] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.712781 11337 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:17.713218 11337 tablet_bootstrap.cc:492] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:15:17.713443 11337 ts_tablet_manager.cc:1367] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.002s
I20240503 14:15:17.713147 10983 raft_consensus.cc:2932] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:17.714152 11337 raft_consensus.cc:348] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:17.714380 11337 raft_consensus.cc:729] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:15:17.714622 11337 consensus_queue.cc:260] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:17.715122 11337 ts_tablet_manager.cc:1398] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:17.715898 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-4d4c447b816b430caf4b213f896e51e3 received from {username='slave'} at 127.20.240.66:46535
I20240503 14:15:17.715999 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session 6409961ee8c0445687c11724213c44a9-4d4c447b816b430caf4b213f896e51e3 on tablet 4d4c447b816b430caf4b213f896e51e3 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:15:17.719693 31201 raft_consensus.cc:2932] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:17.726286 27899 catalog_manager.cc:5484] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:17.728626 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(dc9c29880eea4cd9a11eb24eeb3bd228): perf score=0.033335
I20240503 14:15:17.731696 30939 raft_consensus.cc:2932] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:17.755004 11312 tablet_copy_client.cc:308] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:15:17.755748 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 529c250699cb4e09bae6129d138e37c2 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:32803)
I20240503 14:15:17.755941 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 529c250699cb4e09bae6129d138e37c2 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:32803: session id = 530893a961af4b95b6b687fa5ed22f9d-529c250699cb4e09bae6129d138e37c2
I20240503 14:15:17.760704 28063 tablet_copy_source_session.cc:205] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:15:17.762850 11312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 529c250699cb4e09bae6129d138e37c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:17.768411 11367 ts_tablet_manager.cc:902] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:17.769804 11312 tablet_copy_client.cc:783] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 5 data blocks...
I20240503 14:15:17.792801 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(dc9c29880eea4cd9a11eb24eeb3bd228) complete. Timing: real 0.064s user 0.039s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":359,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1512,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1089,"threads_started":2,"wal-append.queue_time_us":929}
I20240503 14:15:17.795673 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(dc9c29880eea4cd9a11eb24eeb3bd228): 4923 bytes on disk
I20240503 14:15:17.796191 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(dc9c29880eea4cd9a11eb24eeb3bd228) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240503 14:15:17.798146 11312 tablet_copy_client.cc:647] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:17.796303 11367 tablet_copy_client.cc:308] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:15:17.799892 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 05a67063845044eea2d56c34cdfd4ab1 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:50515)
I20240503 14:15:17.800019 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 05a67063845044eea2d56c34cdfd4ab1 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:50515: session id = 6bbc686598f24f518d7ca6bb2063058d-05a67063845044eea2d56c34cdfd4ab1
I20240503 14:15:17.800839 11312 tablet_copy_client.cc:523] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:17.803474 28454 tablet_copy_source_session.cc:205] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 5 blocks and 1 log segments
I20240503 14:15:17.804831 11312 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:15:17.813536 11337 ts_tablet_manager.cc:902] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:17.815562 11337 tablet_copy_client.cc:308] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:15:17.816280 11367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a67063845044eea2d56c34cdfd4ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:17.816370 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet fbd2cac7ae8e45b6bd407b119efe3a51 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:41995)
I20240503 14:15:17.816488 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet fbd2cac7ae8e45b6bd407b119efe3a51 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:41995: session id = 6409961ee8c0445687c11724213c44a9-fbd2cac7ae8e45b6bd407b119efe3a51
I20240503 14:15:17.819260 28063 tablet_copy_source_session.cc:205] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:17.820354 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(664cadf5232f4768adf6438e57f7b7c9): perf score=0.033357
I20240503 14:15:17.822222 11337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd2cac7ae8e45b6bd407b119efe3a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:17.826004 11367 tablet_copy_client.cc:783] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 5 data blocks...
I20240503 14:15:17.834964 11312 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:17.835377 11312 tablet_bootstrap.cc:492] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:15:17.835641 11312 ts_tablet_manager.cc:1367] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.001s
I20240503 14:15:17.836329 11312 raft_consensus.cc:348] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:17.836534 11312 raft_consensus.cc:729] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:15:17.836767 11312 consensus_queue.cc:260] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:17.837292 11312 ts_tablet_manager.cc:1398] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:17.837431 11337 tablet_copy_client.cc:783] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:17.838093 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-529c250699cb4e09bae6129d138e37c2 received from {username='slave'} at 127.20.240.72:32803
I20240503 14:15:17.838212 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-529c250699cb4e09bae6129d138e37c2 on tablet 529c250699cb4e09bae6129d138e37c2 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:15:17.837674 11337 tablet_copy_client.cc:647] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:17.840764 11337 tablet_copy_client.cc:523] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:17.843499 11337 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:15:17.854620 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(dc9c29880eea4cd9a11eb24eeb3bd228): perf score=0.033363
I20240503 14:15:17.855230 11337 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:17.855542 11337 tablet_bootstrap.cc:492] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:15:17.855758 11337 ts_tablet_manager.cc:1367] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 14:15:17.856577 11337 raft_consensus.cc:348] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:17.856799 11337 raft_consensus.cc:729] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:15:17.857028 11337 consensus_queue.cc:260] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:17.857668 11337 ts_tablet_manager.cc:1398] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:17.860028 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-fbd2cac7ae8e45b6bd407b119efe3a51 received from {username='slave'} at 127.20.240.66:41995
I20240503 14:15:17.860133 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 6409961ee8c0445687c11724213c44a9-fbd2cac7ae8e45b6bd407b119efe3a51 on tablet fbd2cac7ae8e45b6bd407b119efe3a51 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:15:17.874043 11367 tablet_copy_client.cc:647] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:17.877488 11367 tablet_copy_client.cc:523] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:17.883438 11367 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:15:17.928682 11394 ts_tablet_manager.cc:902] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:17.931864 30939 consensus_queue.cc:237] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:17.933951 28043 raft_consensus.cc:1203] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 14:15:17.934940 30875 consensus_peers.cc:485] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b4544aefcc54a828dc3b75726367f70. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:17.939152 28433 raft_consensus.cc:1261] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.939965 11219 consensus_queue.cc:1035] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.940690 28302 raft_consensus.cc:1261] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:17.946548 11047 consensus_queue.cc:1035] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:17.961450 11337 ts_tablet_manager.cc:902] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:17.988042 11367 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:17.988629 11367 tablet_bootstrap.cc:492] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:15:17.989655 11367 ts_tablet_manager.cc:1367] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.106s user 0.019s sys 0.005s
I20240503 14:15:17.993165 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(d7ba7409f5bd4531bc5370019aca14c7): perf score=0.033399
I20240503 14:15:17.998310 31200 raft_consensus.cc:1203] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:17.997637 11367 raft_consensus.cc:348] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:17.999938 11367 raft_consensus.cc:729] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.000211 11367 consensus_queue.cc:260] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:18.001758 11219 raft_consensus.cc:2932] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:18.007728 11337 tablet_copy_client.cc:308] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.68:38371
I20240503 14:15:18.007530 28302 raft_consensus.cc:2932] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:18.008332 28454 tablet_copy_service.cc:140] P 5388902973134958a030fd63ab4faba4: Received BeginTabletCopySession request for tablet 130a2a0b5aa344d3be535cfde8990d39 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:55357)
I20240503 14:15:18.008473 28454 tablet_copy_service.cc:161] P 5388902973134958a030fd63ab4faba4: Beginning new tablet copy session on tablet 130a2a0b5aa344d3be535cfde8990d39 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:55357: session id = 6409961ee8c0445687c11724213c44a9-130a2a0b5aa344d3be535cfde8990d39
I20240503 14:15:18.011202 28454 tablet_copy_source_session.cc:205] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:18.013566 11337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 130a2a0b5aa344d3be535cfde8990d39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.017536 27893 catalog_manager.cc:5484] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:18.018643 28434 raft_consensus.cc:2932] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:18.029165 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(dc9c29880eea4cd9a11eb24eeb3bd228) complete. Timing: real 0.174s user 0.037s sys 0.026s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":324,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":3376,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":802,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71037,"threads_started":2,"wal-append.queue_time_us":67986}
I20240503 14:15:18.032318 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(dc9c29880eea4cd9a11eb24eeb3bd228): 4923 bytes on disk
I20240503 14:15:18.000869 11367 ts_tablet_manager.cc:1398] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:15:18.037521 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(dc9c29880eea4cd9a11eb24eeb3bd228) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
I20240503 14:15:18.037629 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-05a67063845044eea2d56c34cdfd4ab1 received from {username='slave'} at 127.20.240.70:50515
I20240503 14:15:18.037722 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-05a67063845044eea2d56c34cdfd4ab1 on tablet 05a67063845044eea2d56c34cdfd4ab1 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:18.039225 11337 tablet_copy_client.cc:783] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:18.039433 11337 tablet_copy_client.cc:647] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:18.040854 11337 tablet_copy_client.cc:523] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:18.045292 11394 tablet_copy_client.cc:308] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:15:18.045756 11337 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:15:18.045893 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet 06e418c4c2cf46608ba454d48fae9285 from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:53677)
I20240503 14:15:18.046000 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet 06e418c4c2cf46608ba454d48fae9285 from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:53677: session id = a4cdf05256a0407c8929ac43eb55d44f-06e418c4c2cf46608ba454d48fae9285
I20240503 14:15:18.048686 28063 tablet_copy_source_session.cc:205] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:18.052218 11337 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:18.052577 11337 tablet_bootstrap.cc:492] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:15:18.052807 11394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e418c4c2cf46608ba454d48fae9285. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.052839 11337 ts_tablet_manager.cc:1367] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 14:15:18.053745 11337 raft_consensus.cc:348] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:18.053992 11337 raft_consensus.cc:729] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.054265 11337 consensus_queue.cc:260] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:18.055084 11337 ts_tablet_manager.cc:1398] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:18.055876 28454 tablet_copy_service.cc:342] P 5388902973134958a030fd63ab4faba4: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-130a2a0b5aa344d3be535cfde8990d39 received from {username='slave'} at 127.20.240.66:55357
I20240503 14:15:18.056012 28454 tablet_copy_service.cc:433] P 5388902973134958a030fd63ab4faba4: ending tablet copy session 6409961ee8c0445687c11724213c44a9-130a2a0b5aa344d3be535cfde8990d39 on tablet 130a2a0b5aa344d3be535cfde8990d39 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:15:18.058477 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(664cadf5232f4768adf6438e57f7b7c9) complete. Timing: real 0.238s user 0.058s sys 0.056s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":17466,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1581,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3695,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":126294,"threads_started":2,"wal-append.queue_time_us":107813}
I20240503 14:15:18.061030 11394 tablet_copy_client.cc:783] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:18.061239 11394 tablet_copy_client.cc:647] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:18.062206 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(664cadf5232f4768adf6438e57f7b7c9): 4838 bytes on disk
I20240503 14:15:18.063277 11394 tablet_copy_client.cc:523] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:18.067569 11394 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:15:18.063273 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(664cadf5232f4768adf6438e57f7b7c9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240503 14:15:18.072002 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(d7ba7409f5bd4531bc5370019aca14c7): perf score=0.033418
I20240503 14:15:18.082679 31249 maintenance_manager.cc:419] P 530893a961af4b95b6b687fa5ed22f9d: Scheduling UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2): 4838 bytes on disk
I20240503 14:15:18.084542 31140 maintenance_manager.cc:643] P 530893a961af4b95b6b687fa5ed22f9d: UndoDeltaBlockGCOp(529c250699cb4e09bae6129d138e37c2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240503 14:15:18.093677 11394 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:18.094045 11394 tablet_bootstrap.cc:492] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:15:18.094336 11394 ts_tablet_manager.cc:1367] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.002s
I20240503 14:15:18.095391 11394 raft_consensus.cc:348] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:18.096006 11394 raft_consensus.cc:729] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.096249 11394 consensus_queue.cc:260] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:18.096810 11394 ts_tablet_manager.cc:1398] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:18.097000 28433 raft_consensus.cc:1203] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:18.100488 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-06e418c4c2cf46608ba454d48fae9285 received from {username='slave'} at 127.20.240.67:53677
I20240503 14:15:18.100593 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-06e418c4c2cf46608ba454d48fae9285 on tablet 06e418c4c2cf46608ba454d48fae9285 with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:18.143810 31200 raft_consensus.cc:1203] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:18.179616 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(d7ba7409f5bd4531bc5370019aca14c7) complete. Timing: real 0.186s user 0.063s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8382,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":12967,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2179,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28536,"threads_started":2,"wal-append.queue_time_us":17956}
I20240503 14:15:18.183992 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(d7ba7409f5bd4531bc5370019aca14c7): 4583 bytes on disk
I20240503 14:15:18.185412 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(d7ba7409f5bd4531bc5370019aca14c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240503 14:15:18.188163 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(664cadf5232f4768adf6438e57f7b7c9): perf score=0.033451
I20240503 14:15:18.188371 30939 raft_consensus.cc:1203] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:15:18.197384 28173 raft_consensus.cc:1203] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:18.205765 28173 raft_consensus.cc:1203] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:18.204993 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(d7ba7409f5bd4531bc5370019aca14c7) complete. Timing: real 0.132s user 0.052s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":342,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1269,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":912,"threads_started":1,"wal-append.queue_time_us":307}
I20240503 14:15:18.209805 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(d7ba7409f5bd4531bc5370019aca14c7): 4583 bytes on disk
I20240503 14:15:18.211562 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(d7ba7409f5bd4531bc5370019aca14c7) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":815,"lbm_reads_lt_1ms":4}
I20240503 14:15:18.219743 30987 maintenance_manager.cc:419] P 6bbc686598f24f518d7ca6bb2063058d: Scheduling UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1): 4923 bytes on disk
I20240503 14:15:18.221103 30878 maintenance_manager.cc:643] P 6bbc686598f24f518d7ca6bb2063058d: UndoDeltaBlockGCOp(05a67063845044eea2d56c34cdfd4ab1) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 14:15:18.265972 11424 ts_tablet_manager.cc:902] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:18.278313 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(664cadf5232f4768adf6438e57f7b7c9) complete. Timing: real 0.089s user 0.039s sys 0.033s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":92,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1403,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":828,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":29314,"threads_started":1,"wal-append.queue_time_us":29390}
I20240503 14:15:18.281953 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(664cadf5232f4768adf6438e57f7b7c9): 4838 bytes on disk
I20240503 14:15:18.282773 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(664cadf5232f4768adf6438e57f7b7c9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240503 14:15:18.294914 11424 tablet_copy_client.cc:308] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.65:33293
I20240503 14:15:18.295554 28063 tablet_copy_service.cc:140] P 793a0d5dd1634c5faaf0e1e4851e8294: Received BeginTabletCopySession request for tablet ff8d8c22d99c462b931a77297c5ff1ff from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:35861)
I20240503 14:15:18.295699 28063 tablet_copy_service.cc:161] P 793a0d5dd1634c5faaf0e1e4851e8294: Beginning new tablet copy session on tablet ff8d8c22d99c462b931a77297c5ff1ff from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:35861: session id = 5388902973134958a030fd63ab4faba4-ff8d8c22d99c462b931a77297c5ff1ff
I20240503 14:15:18.297590 28063 tablet_copy_source_session.cc:205] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:18.299495 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff8d8c22d99c462b931a77297c5ff1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.307936 11424 tablet_copy_client.cc:783] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:18.308182 11424 tablet_copy_client.cc:647] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:18.309573 11424 tablet_copy_client.cc:523] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:18.314438 11424 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:15:18.321588 11424 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:18.322130 11424 tablet_bootstrap.cc:492] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:15:18.322423 11424 ts_tablet_manager.cc:1367] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 14:15:18.323097 11424 raft_consensus.cc:348] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:18.323336 11424 raft_consensus.cc:729] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.323536 11424 consensus_queue.cc:260] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:18.324066 11424 ts_tablet_manager.cc:1398] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:18.325812 28063 tablet_copy_service.cc:342] P 793a0d5dd1634c5faaf0e1e4851e8294: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-ff8d8c22d99c462b931a77297c5ff1ff received from {username='slave'} at 127.20.240.68:35861
I20240503 14:15:18.325909 28063 tablet_copy_service.cc:433] P 793a0d5dd1634c5faaf0e1e4851e8294: ending tablet copy session 5388902973134958a030fd63ab4faba4-ff8d8c22d99c462b931a77297c5ff1ff on tablet ff8d8c22d99c462b931a77297c5ff1ff with peer 5388902973134958a030fd63ab4faba4
I20240503 14:15:18.377492 28173 raft_consensus.cc:1203] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:18.427028 11451 ts_tablet_manager.cc:902] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:18.538000 28303 raft_consensus.cc:1203] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:18.618789 11451 tablet_copy_client.cc:308] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:15:18.619622 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 1b4544aefcc54a828dc3b75726367f70 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:53615)
I20240503 14:15:18.619786 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 1b4544aefcc54a828dc3b75726367f70 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:53615: session id = 530893a961af4b95b6b687fa5ed22f9d-1b4544aefcc54a828dc3b75726367f70
I20240503 14:15:18.622519 30961 tablet_copy_source_session.cc:205] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:18.624794 11451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b4544aefcc54a828dc3b75726367f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:18.645630 11451 tablet_copy_client.cc:783] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:18.645993 11451 tablet_copy_client.cc:647] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:18.653090 11451 tablet_copy_client.cc:523] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:18.669080 11451 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:15:18.699694 11451 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:18.700058 11451 tablet_bootstrap.cc:492] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:15:18.700320 11451 ts_tablet_manager.cc:1367] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.004s
I20240503 14:15:18.701090 11451 raft_consensus.cc:348] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:18.701354 11451 raft_consensus.cc:729] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:15:18.701697 11451 consensus_queue.cc:260] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:18.702329 11451 ts_tablet_manager.cc:1398] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:18.707844 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-1b4544aefcc54a828dc3b75726367f70 received from {username='slave'} at 127.20.240.72:53615
I20240503 14:15:18.707939 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-1b4544aefcc54a828dc3b75726367f70 on tablet 1b4544aefcc54a828dc3b75726367f70 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:15:18.799535 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:18.811575 28433 raft_consensus.cc:1203] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:18.815819 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:18.826295 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:18.843812 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:18.858803 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:18.859516 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:18.865818 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:15:18.871083 28108 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.72:41331, user_credentials={real_user=slave}} blocked reactor thread for 51391.9us
I20240503 14:15:18.894793 31201 raft_consensus.cc:1203] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:18.919032 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:19.391753 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:19.410475 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:19.469084 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:19.503039 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:19.524191 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:19.543586 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:19.561326 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:19.571063 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:19.726634 28043 consensus_queue.cc:237] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:19.728190 28433 raft_consensus.cc:1261] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:19.728209 31201 raft_consensus.cc:1261] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:19.728686 11543 consensus_queue.cc:1035] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:19.728945 10983 consensus_queue.cc:1035] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:19.731459 10983 raft_consensus.cc:2932] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:19.735595 28433 raft_consensus.cc:2932] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:19.740324 27896 catalog_manager.cc:5484] T ff8d8c22d99c462b931a77297c5ff1ff P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:19.742692 31201 raft_consensus.cc:2932] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:19.758164 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet ff8d8c22d99c462b931a77297c5ff1ff with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:15:19.764501 11596 tablet_replica.cc:330] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:15:19.764883 11596 raft_consensus.cc:2229] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:19.765082 11596 raft_consensus.cc:2258] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:19.765731 11596 ts_tablet_manager.cc:1875] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:19.771564 11596 ts_tablet_manager.cc:1888] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:19.771693 11596 log.cc:1196] T ff8d8c22d99c462b931a77297c5ff1ff P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/ff8d8c22d99c462b931a77297c5ff1ff
I20240503 14:15:19.772465 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet ff8d8c22d99c462b931a77297c5ff1ff (table rebalancer_extra_table_2 [id=ceeff6e8ae044b7c8fc0d7663df1b1a9]) successfully deleted
I20240503 14:15:20.033288 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:20.034530 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:20.039073 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:20.049580 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:20.087090 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:20.105430 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:20.143213 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:20.175638 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:20.324863 28432 consensus_queue.cc:237] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:20.327539 28173 raft_consensus.cc:1261] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:20.327705 30807 raft_consensus.cc:1261] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:20.328167 11256 consensus_queue.cc:1035] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:20.328395 10784 consensus_queue.cc:1035] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:20.331521 10784 raft_consensus.cc:2932] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:20.333608 30807 raft_consensus.cc:2932] T 130a2a0b5aa344d3be535cfde8990d39 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:20.341975 27893 catalog_manager.cc:5484] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 1 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:20.348035 28171 raft_consensus.cc:2932] T 130a2a0b5aa344d3be535cfde8990d39 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:20.358824 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet 130a2a0b5aa344d3be535cfde8990d39 with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52616
I20240503 14:15:20.360074 11678 tablet_replica.cc:330] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:15:20.360383 11678 raft_consensus.cc:2229] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:20.362123 11678 raft_consensus.cc:2258] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:20.362888 11678 ts_tablet_manager.cc:1875] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:20.369577 11678 ts_tablet_manager.cc:1888] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:20.369714 11678 log.cc:1196] T 130a2a0b5aa344d3be535cfde8990d39 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/130a2a0b5aa344d3be535cfde8990d39
I20240503 14:15:20.370489 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet 130a2a0b5aa344d3be535cfde8990d39 (table rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]) successfully deleted
I20240503 14:15:20.768440 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:20.776141 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:20.808811 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:20.856391 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:20.856496 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:21.004940 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(48266bf4ad9b4976b639553490ffebbe): perf score=0.033345
I20240503 14:15:21.011610 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:21.021184 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:21.036060 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:21.074600 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(cb88c79b4b1a40c2b45c2dd79ab894d4): perf score=0.033362
I20240503 14:15:21.091637 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(cd84de9286eb4ad7b88b9f243eb2801c): perf score=0.033362
I20240503 14:15:21.100646 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(cb88c79b4b1a40c2b45c2dd79ab894d4): perf score=0.033370
I20240503 14:15:21.118148 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(cb88c79b4b1a40c2b45c2dd79ab894d4) complete. Timing: real 0.043s user 0.038s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6318,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":11474,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":739,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":6886,"threads_started":2,"wal-append.queue_time_us":327}
I20240503 14:15:21.122218 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(cb88c79b4b1a40c2b45c2dd79ab894d4): 4838 bytes on disk
I20240503 14:15:21.123239 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(cb88c79b4b1a40c2b45c2dd79ab894d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 14:15:21.265753 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(48266bf4ad9b4976b639553490ffebbe) complete. Timing: real 0.261s user 0.080s sys 0.042s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56959,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":4876,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":64974,"threads_started":2,"wal-append.queue_time_us":8061}
I20240503 14:15:21.270000 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(48266bf4ad9b4976b639553490ffebbe): 4923 bytes on disk
I20240503 14:15:21.272768 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(48266bf4ad9b4976b639553490ffebbe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
I20240503 14:15:21.279834 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(cd84de9286eb4ad7b88b9f243eb2801c): perf score=0.033418
I20240503 14:15:21.307901 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(cd84de9286eb4ad7b88b9f243eb2801c) complete. Timing: real 0.216s user 0.036s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":22538,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":7412,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":787,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":86313,"threads_started":2,"wal-append.queue_time_us":53652}
I20240503 14:15:21.318979 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(cd84de9286eb4ad7b88b9f243eb2801c): 4583 bytes on disk
I20240503 14:15:21.319741 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(cd84de9286eb4ad7b88b9f243eb2801c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 14:15:21.332110 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(cb88c79b4b1a40c2b45c2dd79ab894d4) complete. Timing: real 0.231s user 0.060s sys 0.032s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":27467,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":15365,"drs_written":1,"lbm_read_time_us":6065,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1383,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":130977,"threads_started":2,"wal-append.queue_time_us":103490}
I20240503 14:15:21.335028 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(cb88c79b4b1a40c2b45c2dd79ab894d4): 4838 bytes on disk
I20240503 14:15:21.335721 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(cb88c79b4b1a40c2b45c2dd79ab894d4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 14:15:21.340382 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(48266bf4ad9b4976b639553490ffebbe): perf score=0.033430
I20240503 14:15:21.399659 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(cd84de9286eb4ad7b88b9f243eb2801c) complete. Timing: real 0.119s user 0.043s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1357,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4123,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4473,"threads_started":1,"wal-append.queue_time_us":4361}
I20240503 14:15:21.403872 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(cd84de9286eb4ad7b88b9f243eb2801c): 4583 bytes on disk
I20240503 14:15:21.411412 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(cd84de9286eb4ad7b88b9f243eb2801c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240503 14:15:21.491652 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(48266bf4ad9b4976b639553490ffebbe) complete. Timing: real 0.151s user 0.038s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":25010,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1400,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2290,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7142,"threads_started":1,"wal-append.queue_time_us":6979}
I20240503 14:15:21.493935 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(48266bf4ad9b4976b639553490ffebbe): 4923 bytes on disk
I20240503 14:15:21.494683 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(48266bf4ad9b4976b639553490ffebbe) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":195,"lbm_reads_lt_1ms":4}
I20240503 14:15:21.538969 28040 consensus_queue.cc:237] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:21.541337 31201 raft_consensus.cc:1261] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:21.541366 28303 raft_consensus.cc:1261] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:21.541879 10662 consensus_queue.cc:1035] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:21.543689 10662 consensus_queue.cc:1035] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:21.546646 11543 raft_consensus.cc:2932] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:21.549584 28303 raft_consensus.cc:2932] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:21.556391 27896 catalog_manager.cc:5484] T 06e418c4c2cf46608ba454d48fae9285 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:21.577908 31200 raft_consensus.cc:2932] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:21.599378 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 06e418c4c2cf46608ba454d48fae9285 with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:15:21.604822 11786 tablet_replica.cc:330] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:15:21.605089 11786 raft_consensus.cc:2229] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:21.605288 11786 raft_consensus.cc:2258] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:21.606010 11786 ts_tablet_manager.cc:1875] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:21.611517 11786 ts_tablet_manager.cc:1888] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:21.611747 11786 log.cc:1196] T 06e418c4c2cf46608ba454d48fae9285 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/06e418c4c2cf46608ba454d48fae9285
I20240503 14:15:21.612509 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 06e418c4c2cf46608ba454d48fae9285 (table rebalancer_extra_table_24 [id=3bd827b8fe4647d9a22fbbb219363b87]) successfully deleted
I20240503 14:15:21.873489 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:21.961913 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:21.986490 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:21.989162 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:22.001529 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:22.012326 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:22.013633 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:22.053560 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:22.203572 28040 consensus_queue.cc:237] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:22.205947 28173 raft_consensus.cc:1261] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:22.206076 30810 raft_consensus.cc:1261] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:22.206498 11777 consensus_queue.cc:1035] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:22.207185 11777 consensus_queue.cc:1035] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:22.209347 11543 raft_consensus.cc:2932] T fbd2cac7ae8e45b6bd407b119efe3a51 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:22.209930 30807 raft_consensus.cc:2932] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:22.210007 28173 raft_consensus.cc:2932] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:22.216725 27896 catalog_manager.cc:5484] T fbd2cac7ae8e45b6bd407b119efe3a51 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:22.248214 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd2cac7ae8e45b6bd407b119efe3a51 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:15:22.263214 11876 tablet_replica.cc:330] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:15:22.263432 11876 raft_consensus.cc:2229] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:22.263612 11876 raft_consensus.cc:2258] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:22.264286 11876 ts_tablet_manager.cc:1875] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:22.274998 11876 ts_tablet_manager.cc:1888] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:22.275143 11876 log.cc:1196] T fbd2cac7ae8e45b6bd407b119efe3a51 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/fbd2cac7ae8e45b6bd407b119efe3a51
I20240503 14:15:22.275873 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet fbd2cac7ae8e45b6bd407b119efe3a51 (table rebalancer_extra_table_4 [id=b94cc8628cea482fb2d93d0350b487e3]) successfully deleted
I20240503 14:15:22.853452 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:22.867653 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:22.904883 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:22.922745 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:22.940021 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:22.983144 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:23.038486 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:23.045753 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:23.160475 28043 consensus_queue.cc:237] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:23.162309 28172 raft_consensus.cc:1261] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:23.162374 31201 raft_consensus.cc:1261] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:15:23.162881 11867 consensus_queue.cc:1035] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:23.163218 11777 consensus_queue.cc:1035] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:23.165187 11867 raft_consensus.cc:2932] T 529c250699cb4e09bae6129d138e37c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:23.167783 31201 raft_consensus.cc:2932] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:23.168591 28172 raft_consensus.cc:2932] T 529c250699cb4e09bae6129d138e37c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:23.174453 27899 catalog_manager.cc:5484] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index 4 to 5, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:23.190191 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 529c250699cb4e09bae6129d138e37c2 with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43626
I20240503 14:15:23.190903 11981 tablet_replica.cc:330] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:15:23.191251 11981 raft_consensus.cc:2229] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:23.191586 11981 raft_consensus.cc:2258] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:23.192386 11981 ts_tablet_manager.cc:1875] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:23.212569 11981 ts_tablet_manager.cc:1888] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:15:23.212715 11981 log.cc:1196] T 529c250699cb4e09bae6129d138e37c2 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/529c250699cb4e09bae6129d138e37c2
I20240503 14:15:23.213480 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 529c250699cb4e09bae6129d138e37c2 (table rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]) successfully deleted
I20240503 14:15:23.357450 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling FlushMRSOp(0225203f2de2498eb448e066ce545a64): perf score=0.033352
I20240503 14:15:23.403796 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling FlushMRSOp(0225203f2de2498eb448e066ce545a64): perf score=0.033367
I20240503 14:15:23.451169 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(edd300244be74e3aa94d7ae35deeb470): perf score=0.033383
I20240503 14:15:23.473230 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: FlushMRSOp(0225203f2de2498eb448e066ce545a64) complete. Timing: real 0.115s user 0.056s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":23155,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":3927,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":928,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":23255,"threads_started":2,"wal-append.queue_time_us":333}
I20240503 14:15:23.476161 28220 maintenance_manager.cc:419] P 6409961ee8c0445687c11724213c44a9: Scheduling UndoDeltaBlockGCOp(0225203f2de2498eb448e066ce545a64): 4923 bytes on disk
I20240503 14:15:23.476722 28111 maintenance_manager.cc:643] P 6409961ee8c0445687c11724213c44a9: UndoDeltaBlockGCOp(0225203f2de2498eb448e066ce545a64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.520525 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(823dde634aac449ea8cd21d38ecfb22b): perf score=0.033396
W20240503 14:15:23.570427 28237 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 63819.2us
I20240503 14:15:23.578523 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:23.585505 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:23.593726 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: FlushMRSOp(0225203f2de2498eb448e066ce545a64) complete. Timing: real 0.185s user 0.045s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14176,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":6227,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":854,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":29265,"threads_started":2,"wal-append.queue_time_us":14886}
I20240503 14:15:23.596740 28089 maintenance_manager.cc:419] P 793a0d5dd1634c5faaf0e1e4851e8294: Scheduling UndoDeltaBlockGCOp(0225203f2de2498eb448e066ce545a64): 4923 bytes on disk
I20240503 14:15:23.598829 27980 maintenance_manager.cc:643] P 793a0d5dd1634c5faaf0e1e4851e8294: UndoDeltaBlockGCOp(0225203f2de2498eb448e066ce545a64) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.605921 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:23.619205 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:23.634804 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:23.659147 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:23.732971 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(edd300244be74e3aa94d7ae35deeb470) complete. Timing: real 0.281s user 0.058s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1585,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1332,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":78313,"threads_started":1,"wal-append.queue_time_us":78188}
I20240503 14:15:23.755309 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(edd300244be74e3aa94d7ae35deeb470): 4836 bytes on disk
I20240503 14:15:23.762190 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:23.770614 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(edd300244be74e3aa94d7ae35deeb470) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.785305 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:23.800756 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling FlushMRSOp(823dde634aac449ea8cd21d38ecfb22b): perf score=0.033469
I20240503 14:15:23.889173 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(823dde634aac449ea8cd21d38ecfb22b) complete. Timing: real 0.368s user 0.048s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4420,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4576,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":29805,"threads_started":2,"wal-append.queue_time_us":23456}
I20240503 14:15:23.899166 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling FlushMRSOp(edd300244be74e3aa94d7ae35deeb470): perf score=0.033498
I20240503 14:15:23.953802 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: FlushMRSOp(823dde634aac449ea8cd21d38ecfb22b) complete. Timing: real 0.153s user 0.033s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1456,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2509,"threads_started":1,"wal-append.queue_time_us":354}
I20240503 14:15:23.956074 28350 maintenance_manager.cc:419] P a4cdf05256a0407c8929ac43eb55d44f: Scheduling UndoDeltaBlockGCOp(823dde634aac449ea8cd21d38ecfb22b): 4583 bytes on disk
I20240503 14:15:23.956583 28241 maintenance_manager.cc:643] P a4cdf05256a0407c8929ac43eb55d44f: UndoDeltaBlockGCOp(823dde634aac449ea8cd21d38ecfb22b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.996282 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: FlushMRSOp(edd300244be74e3aa94d7ae35deeb470) complete. Timing: real 0.082s user 0.053s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1365,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":209,"threads_started":1,"wal-append.queue_time_us":289}
I20240503 14:15:23.998689 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(edd300244be74e3aa94d7ae35deeb470): 4836 bytes on disk
I20240503 14:15:23.999265 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(edd300244be74e3aa94d7ae35deeb470) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 14:15:24.001673 28480 maintenance_manager.cc:419] P 5388902973134958a030fd63ab4faba4: Scheduling UndoDeltaBlockGCOp(823dde634aac449ea8cd21d38ecfb22b): 4583 bytes on disk
I20240503 14:15:24.002115 28371 maintenance_manager.cc:643] P 5388902973134958a030fd63ab4faba4: UndoDeltaBlockGCOp(823dde634aac449ea8cd21d38ecfb22b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240503 14:15:24.012106 30940 consensus_queue.cc:237] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:24.013944 31201 raft_consensus.cc:1261] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:24.013978 28043 raft_consensus.cc:1261] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:24.014467 11780 consensus_queue.cc:1035] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:24.014912 11780 consensus_queue.cc:1035] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:24.016853 11780 raft_consensus.cc:2932] T 72c7ff14d43342439f935389b03ecad0 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:24.017699 28043 raft_consensus.cc:2932] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:24.018553 31201 raft_consensus.cc:2932] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:24.024960 27896 catalog_manager.cc:5484] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:24.038226 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 72c7ff14d43342439f935389b03ecad0 with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:15:24.038812 12077 tablet_replica.cc:330] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:15:24.039038 12077 raft_consensus.cc:2229] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:24.039294 12077 raft_consensus.cc:2258] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:24.039940 12077 ts_tablet_manager.cc:1875] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:24.045447 12077 ts_tablet_manager.cc:1888] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:24.045589 12077 log.cc:1196] T 72c7ff14d43342439f935389b03ecad0 P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/72c7ff14d43342439f935389b03ecad0
I20240503 14:15:24.046808 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 72c7ff14d43342439f935389b03ecad0 (table rebalancer_extra_table_0 [id=c0a512d67b684dadb3f576aa0cfb4740]) successfully deleted
I20240503 14:15:24.345578 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:24.358752 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:24.375653 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:24.386873 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:24.394480 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:24.481745 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:15:24.484890 28237 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.69:39689, user_credentials={real_user=slave}} blocked reactor thread for 45014.6us
I20240503 14:15:24.516433 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:24.524926 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:24.761338 30810 consensus_queue.cc:237] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:24.763581 30941 raft_consensus.cc:1261] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:24.763718 28173 raft_consensus.cc:1261] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:24.764191 11181 consensus_queue.cc:1035] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:24.764809 11181 consensus_queue.cc:1035] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:24.771646 11720 raft_consensus.cc:2932] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:24.772339 30941 raft_consensus.cc:2932] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:24.779825 28173 raft_consensus.cc:2932] T 4d4c447b816b430caf4b213f896e51e3 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:24.780198 27897 catalog_manager.cc:5484] T 4d4c447b816b430caf4b213f896e51e3 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index 2 to 3, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:24.794677 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d4c447b816b430caf4b213f896e51e3 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:45512
I20240503 14:15:24.795352 12163 tablet_replica.cc:330] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:15:24.795665 12163 raft_consensus.cc:2229] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:24.795956 12163 raft_consensus.cc:2258] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:24.796586 12163 ts_tablet_manager.cc:1875] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:24.803432 12163 ts_tablet_manager.cc:1888] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:24.803548 12163 log.cc:1196] T 4d4c447b816b430caf4b213f896e51e3 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/4d4c447b816b430caf4b213f896e51e3
I20240503 14:15:24.804438 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 4d4c447b816b430caf4b213f896e51e3 (table rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]) successfully deleted
I20240503 14:15:25.122344 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:25.133116 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:25.148998 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:25.172752 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240503 14:15:25.248502 28368 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 99855.7us
I20240503 14:15:25.283893 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:25.303144 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:25.349753 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:25.366835 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:25.476192 30810 consensus_queue.cc:237] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:25.478053 28304 raft_consensus.cc:1261] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:25.478092 31199 raft_consensus.cc:1261] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:25.478637 12154 consensus_queue.cc:1035] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:25.478919 11181 consensus_queue.cc:1035] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:25.482733 12154 raft_consensus.cc:2932] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:25.486490 28304 raft_consensus.cc:2932] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:25.487066 31199 raft_consensus.cc:2932] T f6151ea340334090a5c0e79ca65af5c1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:25.495625 27893 catalog_manager.cc:5484] T f6151ea340334090a5c0e79ca65af5c1 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:25.529701 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet f6151ea340334090a5c0e79ca65af5c1 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:15:25.535133 12251 tablet_replica.cc:330] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:15:25.538519 12251 raft_consensus.cc:2229] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:25.538829 12251 raft_consensus.cc:2258] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:25.540653 12251 ts_tablet_manager.cc:1875] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:25.547506 12251 ts_tablet_manager.cc:1888] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:25.547695 12251 log.cc:1196] T f6151ea340334090a5c0e79ca65af5c1 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/f6151ea340334090a5c0e79ca65af5c1
I20240503 14:15:25.548512 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet f6151ea340334090a5c0e79ca65af5c1 (table rebalancer_extra_table_36 [id=1e1dcbe104d549689601d3ca207c3e59]) successfully deleted
I20240503 14:15:25.910223 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:25.926909 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:25.961190 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:25.968907 31183 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:15:26.027112 31050 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:26.027576 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:26.082494 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:26.122282 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:26.191141 31070 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5"
dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50936
I20240503 14:15:26.191305 31070 raft_consensus.cc:595] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:26.227167 11922 raft_consensus.cc:979] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: : Instructing follower 530893a961af4b95b6b687fa5ed22f9d to start an election
I20240503 14:15:26.227314 11922 raft_consensus.cc:1067] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Signalling peer 530893a961af4b95b6b687fa5ed22f9d to start an election
I20240503 14:15:26.227980 31201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5"
dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
from {username='slave'} at 127.20.240.71:53445
I20240503 14:15:26.228252 31201 raft_consensus.cc:482] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:26.228341 31201 raft_consensus.cc:3037] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.230813 31201 raft_consensus.cc:504] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:26.232292 31201 leader_election.cc:288] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:26.232496 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:15:26.232686 28172 raft_consensus.cc:3037] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.234629 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:26.234789 28172 raft_consensus.cc:2454] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 2.
I20240503 14:15:26.234824 28433 raft_consensus.cc:3037] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.234967 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1af2fb3fb3fd4dfe873d124232d812b5" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:15:26.235153 31070 raft_consensus.cc:3032] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:26.235229 31070 raft_consensus.cc:729] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:15:26.235460 31070 consensus_queue.cc:260] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:26.235850 31070 raft_consensus.cc:3037] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:26.237252 28433 raft_consensus.cc:2454] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 2.
I20240503 14:15:26.237915 31139 leader_election.cc:302] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters:
I20240503 14:15:26.238209 12323 raft_consensus.cc:2781] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:26.238385 12323 raft_consensus.cc:686] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 2 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:15:26.238687 12323 consensus_queue.cc:237] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:26.240316 31070 raft_consensus.cc:2454] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 2.
I20240503 14:15:26.254442 27899 catalog_manager.cc:5484] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 1 to 2, leader changed from fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 2 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:26.504151 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:26.522905 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:26.543437 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:26.572435 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:26.573637 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:26.582619 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:26.619550 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:26.619939 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:26.796671 31070 raft_consensus.cc:1261] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:26.797309 12323 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:26.822659 28432 raft_consensus.cc:1261] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:26.823529 12323 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:26.829229 28172 raft_consensus.cc:1261] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:26.829738 11540 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:27.107810 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:27.115170 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:27.148291 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:27.150305 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:27.161666 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:27.219530 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:27.220858 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:27.275030 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:27.415024 28303 consensus_queue.cc:237] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:27.418120 31070 raft_consensus.cc:1261] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:27.418211 28172 raft_consensus.cc:1261] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Refusing update from remote peer a4cdf05256a0407c8929ac43eb55d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:27.418658 12129 consensus_queue.cc:1035] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:27.418932 12270 consensus_queue.cc:1035] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:27.422312 12270 raft_consensus.cc:2932] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:27.424000 31069 raft_consensus.cc:2932] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:27.432181 27897 catalog_manager.cc:5484] T 92f507cdcfb04eb4a9eb448773206783 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New cstate: current_term: 1 leader_uuid: "a4cdf05256a0407c8929ac43eb55d44f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:27.438208 28172 raft_consensus.cc:2932] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:27.447098 30921 tablet_service.cc:1514] Processing DeleteTablet for tablet 92f507cdcfb04eb4a9eb448773206783 with delete_type TABLET_DATA_TOMBSTONED (TS 6bbc686598f24f518d7ca6bb2063058d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:54338
I20240503 14:15:27.447908 12493 tablet_replica.cc:330] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: stopping tablet replica
I20240503 14:15:27.448519 12493 raft_consensus.cc:2229] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:27.448755 12493 raft_consensus.cc:2258] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:27.449438 12493 ts_tablet_manager.cc:1875] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:27.457144 12493 ts_tablet_manager.cc:1888] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:27.457355 12493 log.cc:1196] T 92f507cdcfb04eb4a9eb448773206783 P 6bbc686598f24f518d7ca6bb2063058d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/wals/92f507cdcfb04eb4a9eb448773206783
I20240503 14:15:27.458135 27888 catalog_manager.cc:4830] TS 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): tablet 92f507cdcfb04eb4a9eb448773206783 (table rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]) successfully deleted
I20240503 14:15:27.745496 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:27.852036 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:27.898623 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:15:27.952909 28240 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.65:33293, user_credentials={real_user=slave}} blocked reactor thread for 88862.1us
I20240503 14:15:27.969079 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:27.976579 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:28.038658 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:28.056015 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:28.087062 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:28.199501 28040 consensus_queue.cc:237] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:28.201524 30807 raft_consensus.cc:1261] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:28.201653 28173 raft_consensus.cc:1261] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:28.202414 12545 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:28.202708 11867 consensus_queue.cc:1035] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:28.205070 12545 raft_consensus.cc:2932] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:28.205677 28173 raft_consensus.cc:2932] T 88f88e7eb7434450af76714256729887 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:28.212112 30807 raft_consensus.cc:2932] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:28.212214 27899 catalog_manager.cc:5484] T 88f88e7eb7434450af76714256729887 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index 4 to 6, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:28.226850 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 88f88e7eb7434450af76714256729887 with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43626
I20240503 14:15:28.234670 12579 tablet_replica.cc:330] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:15:28.235005 12579 raft_consensus.cc:2229] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:28.235262 12579 raft_consensus.cc:2258] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:28.236056 12579 ts_tablet_manager.cc:1875] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:28.251749 12579 ts_tablet_manager.cc:1888] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:15:28.251962 12579 log.cc:1196] T 88f88e7eb7434450af76714256729887 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/88f88e7eb7434450af76714256729887
I20240503 14:15:28.252825 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 88f88e7eb7434450af76714256729887 (table rebalance_test_table_3 [id=532b45778ec745f6a99ffe981dd1eec5]) successfully deleted
I20240503 14:15:28.626574 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:28.672592 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:28.699466 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:28.712340 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:28.727068 30921 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:28.730487 28412 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:28.731654 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:28.804360 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:28.908601 28433 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1"
dest_uuid: "5388902973134958a030fd63ab4faba4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37412
I20240503 14:15:28.908860 28433 raft_consensus.cc:595] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:15:28.991320 12022 raft_consensus.cc:979] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: : Instructing follower 6bbc686598f24f518d7ca6bb2063058d to start an election
I20240503 14:15:28.991477 12022 raft_consensus.cc:1067] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Signalling peer 6bbc686598f24f518d7ca6bb2063058d to start an election
I20240503 14:15:28.996069 30941 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1"
dest_uuid: "6bbc686598f24f518d7ca6bb2063058d"
from {username='slave'} at 127.20.240.68:36635
I20240503 14:15:28.996490 30941 raft_consensus.cc:482] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:15:28.996670 30941 raft_consensus.cc:3037] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:29.001984 30941 raft_consensus.cc:504] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:29.003846 28304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a4cdf05256a0407c8929ac43eb55d44f"
I20240503 14:15:29.003898 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:15:29.004058 28304 raft_consensus.cc:3037] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:29.004146 28173 raft_consensus.cc:3037] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:29.006268 28304 raft_consensus.cc:2454] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 2.
I20240503 14:15:29.007719 28173 raft_consensus.cc:2454] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 2.
I20240503 14:15:29.008772 30877 leader_election.cc:302] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f; no voters:
I20240503 14:15:29.009163 12388 raft_consensus.cc:2781] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:29.009332 12388 raft_consensus.cc:686] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 2 LEADER]: Becoming Leader. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Running, Role: LEADER
I20240503 14:15:29.009616 12388 consensus_queue.cc:237] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:29.012255 28433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a67063845044eea2d56c34cdfd4ab1" candidate_uuid: "6bbc686598f24f518d7ca6bb2063058d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:29.012466 28433 raft_consensus.cc:3032] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:15:29.012537 28433 raft_consensus.cc:729] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:29.012828 28433 consensus_queue.cc:260] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:29.013235 28433 raft_consensus.cc:3037] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:29.015059 27899 catalog_manager.cc:5484] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: term changed from 1 to 2, leader changed from 5388902973134958a030fd63ab4faba4 (127.20.240.68) to 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70). New cstate: current_term: 2 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:29.016889 28433 raft_consensus.cc:2454] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bbc686598f24f518d7ca6bb2063058d in term 2.
I20240503 14:15:29.017932 30941 leader_election.cc:288] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [CANDIDATE]: Term 2 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:29.259693 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:29.261960 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:29.279328 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:29.290773 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:29.294503 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:29.303218 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:29.361946 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:29.410589 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:29.498646 28432 raft_consensus.cc:1261] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:15:29.502765 12544 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:29.510166 28172 raft_consensus.cc:1261] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:15:29.511005 12544 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:29.541520 28304 raft_consensus.cc:1261] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:15:29.542415 12388 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:15:29.875485 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:29.901464 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:29.906714 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:29.941795 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:29.964593 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:30.020944 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240503 14:15:30.046634 28239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 71204.2us
I20240503 14:15:30.072865 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:30.073709 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:30.274322 28173 consensus_queue.cc:237] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:30.279311 30807 raft_consensus.cc:1261] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:30.279762 30941 raft_consensus.cc:1261] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:30.279889 12643 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:30.280350 12721 consensus_queue.cc:1035] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:30.283025 12643 raft_consensus.cc:2932] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:30.285429 30941 raft_consensus.cc:2932] T f7bdba20d1844eefb6d3b29c5deb2873 P 6bbc686598f24f518d7ca6bb2063058d [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:30.287087 30807 raft_consensus.cc:2932] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:30.293746 27900 catalog_manager.cc:5484] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index 2 to 4, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) evicted. New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:30.302542 28021 tablet_service.cc:1514] Processing DeleteTablet for tablet f7bdba20d1844eefb6d3b29c5deb2873 with delete_type TABLET_DATA_TOMBSTONED (TS 793a0d5dd1634c5faaf0e1e4851e8294 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52616
I20240503 14:15:30.303092 12820 tablet_replica.cc:330] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: stopping tablet replica
I20240503 14:15:30.303331 12820 raft_consensus.cc:2229] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:30.303524 12820 raft_consensus.cc:2258] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:30.304122 12820 ts_tablet_manager.cc:1875] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:30.309440 12820 ts_tablet_manager.cc:1888] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:15:30.309535 12820 log.cc:1196] T f7bdba20d1844eefb6d3b29c5deb2873 P 793a0d5dd1634c5faaf0e1e4851e8294: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7bdba20d1844eefb6d3b29c5deb2873
I20240503 14:15:30.310145 27889 catalog_manager.cc:4830] TS 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): tablet f7bdba20d1844eefb6d3b29c5deb2873 (table rebalancer_extra_table_66 [id=19aeca4f8f854906873a7d51ac0deaf0]) successfully deleted
I20240503 14:15:30.367755 31069 consensus_queue.cc:237] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:30.370358 31201 raft_consensus.cc:1261] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:30.370518 30807 raft_consensus.cc:1261] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:30.370894 31008 consensus_peers.cc:485] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 917de496d0cb4c6d94c6dcf630b90f4a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:30.371493 12650 consensus_queue.cc:1035] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:30.371960 12650 consensus_queue.cc:1035] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:30.380089 11922 raft_consensus.cc:2932] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:30.380590 30807 raft_consensus.cc:2932] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:30.385685 31201 raft_consensus.cc:2932] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:30.388625 27900 catalog_manager.cc:5484] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:30.445001 28173 consensus_queue.cc:237] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:30.448652 31201 raft_consensus.cc:1261] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:30.448648 30941 raft_consensus.cc:1261] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:30.449579 12625 consensus_queue.cc:1035] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:30.449959 12866 consensus_queue.cc:1035] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:30.450310 28110 consensus_peers.cc:485] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 02559d1ab4824298ba76a44a7f05303b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:30.452716 12721 raft_consensus.cc:2932] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:30.453336 30941 raft_consensus.cc:2932] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:30.456344 31201 raft_consensus.cc:2932] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:30.461061 27902 catalog_manager.cc:5484] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:30.496212 28172 consensus_queue.cc:237] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:30.498935 30941 raft_consensus.cc:1261] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:30.499043 30807 raft_consensus.cc:1261] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:30.499670 28108 consensus_peers.cc:485] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 -> Peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): Couldn't send request to peer 530893a961af4b95b6b687fa5ed22f9d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a82470f287274b1c878089fa9d56ef40. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:30.500013 12625 consensus_queue.cc:1035] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:30.500494 12625 consensus_queue.cc:1035] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:30.524168 12866 raft_consensus.cc:2932] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:30.526860 30939 raft_consensus.cc:2932] T a82470f287274b1c878089fa9d56ef40 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:30.528894 30807 raft_consensus.cc:2932] T a82470f287274b1c878089fa9d56ef40 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:30.533615 27900 catalog_manager.cc:5484] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index -1 to 2, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) added. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:30.967111 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:30.991603 12952 ts_tablet_manager.cc:902] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:31.027758 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:31.028645 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:31.049726 12957 ts_tablet_manager.cc:902] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:31.062726 12953 ts_tablet_manager.cc:902] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: Initiating tablet copy from peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:31.073768 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:31.089974 12957 tablet_copy_client.cc:308] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:15:31.091077 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet 02559d1ab4824298ba76a44a7f05303b from peer 5388902973134958a030fd63ab4faba4 ({username='slave'} at 127.20.240.68:42383)
I20240503 14:15:31.091251 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet 02559d1ab4824298ba76a44a7f05303b from peer 5388902973134958a030fd63ab4faba4 at {username='slave'} at 127.20.240.68:42383: session id = 5388902973134958a030fd63ab4faba4-02559d1ab4824298ba76a44a7f05303b
I20240503 14:15:31.099416 28194 tablet_copy_source_session.cc:205] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:31.101676 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02559d1ab4824298ba76a44a7f05303b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:31.125633 12957 tablet_copy_client.cc:783] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:31.126217 12957 tablet_copy_client.cc:647] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:31.128629 12957 tablet_copy_client.cc:523] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:31.145653 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:31.158987 12957 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: Bootstrap starting.
I20240503 14:15:31.171286 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:31.180117 12957 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:31.183015 12957 tablet_bootstrap.cc:492] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: Bootstrap complete.
I20240503 14:15:31.183454 12957 ts_tablet_manager.cc:1367] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.002s
I20240503 14:15:31.186935 12957 raft_consensus.cc:348] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:31.188066 12957 raft_consensus.cc:729] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5388902973134958a030fd63ab4faba4, State: Initialized, Role: FOLLOWER
I20240503 14:15:31.188508 12957 consensus_queue.cc:260] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:31.192358 12957 ts_tablet_manager.cc:1398] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:15:31.193274 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 5388902973134958a030fd63ab4faba4-02559d1ab4824298ba76a44a7f05303b received from {username='slave'} at 127.20.240.68:42383
I20240503 14:15:31.193416 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 5388902973134958a030fd63ab4faba4-02559d1ab4824298ba76a44a7f05303b on tablet 02559d1ab4824298ba76a44a7f05303b with peer 5388902973134958a030fd63ab4faba4
I20240503 14:15:31.211748 12952 tablet_copy_client.cc:308] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:15:31.212399 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet 917de496d0cb4c6d94c6dcf630b90f4a from peer a4cdf05256a0407c8929ac43eb55d44f ({username='slave'} at 127.20.240.67:54103)
I20240503 14:15:31.212529 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet 917de496d0cb4c6d94c6dcf630b90f4a from peer a4cdf05256a0407c8929ac43eb55d44f at {username='slave'} at 127.20.240.67:54103: session id = a4cdf05256a0407c8929ac43eb55d44f-917de496d0cb4c6d94c6dcf630b90f4a
I20240503 14:15:31.212555 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:31.215440 31092 tablet_copy_source_session.cc:205] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:31.216831 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 917de496d0cb4c6d94c6dcf630b90f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:31.222486 12953 tablet_copy_client.cc:308] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.66:37039
I20240503 14:15:31.223114 28194 tablet_copy_service.cc:140] P 6409961ee8c0445687c11724213c44a9: Received BeginTabletCopySession request for tablet a82470f287274b1c878089fa9d56ef40 from peer 530893a961af4b95b6b687fa5ed22f9d ({username='slave'} at 127.20.240.72:59619)
I20240503 14:15:31.223273 28194 tablet_copy_service.cc:161] P 6409961ee8c0445687c11724213c44a9: Beginning new tablet copy session on tablet a82470f287274b1c878089fa9d56ef40 from peer 530893a961af4b95b6b687fa5ed22f9d at {username='slave'} at 127.20.240.72:59619: session id = 530893a961af4b95b6b687fa5ed22f9d-a82470f287274b1c878089fa9d56ef40
I20240503 14:15:31.224169 12952 tablet_copy_client.cc:783] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:31.224452 12952 tablet_copy_client.cc:647] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:31.226500 28194 tablet_copy_source_session.cc:205] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:31.227826 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a82470f287274b1c878089fa9d56ef40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:31.230631 12952 tablet_copy_client.cc:523] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:31.233803 12952 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap starting.
I20240503 14:15:31.241423 12953 tablet_copy_client.cc:783] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:31.241703 12953 tablet_copy_client.cc:647] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:31.251160 12953 tablet_copy_client.cc:523] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:31.251492 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:31.256505 12953 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap starting.
I20240503 14:15:31.286749 12952 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:31.287201 12952 tablet_bootstrap.cc:492] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: Bootstrap complete.
I20240503 14:15:31.287518 12952 ts_tablet_manager.cc:1367] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: Time spent bootstrapping tablet: real 0.054s user 0.018s sys 0.015s
I20240503 14:15:31.288200 12952 raft_consensus.cc:348] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:31.288539 12952 raft_consensus.cc:729] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cdf05256a0407c8929ac43eb55d44f, State: Initialized, Role: FOLLOWER
I20240503 14:15:31.288800 12952 consensus_queue.cc:260] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:31.289958 12952 ts_tablet_manager.cc:1398] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:31.291267 31092 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session a4cdf05256a0407c8929ac43eb55d44f-917de496d0cb4c6d94c6dcf630b90f4a received from {username='slave'} at 127.20.240.67:54103
I20240503 14:15:31.291376 31092 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session a4cdf05256a0407c8929ac43eb55d44f-917de496d0cb4c6d94c6dcf630b90f4a on tablet 917de496d0cb4c6d94c6dcf630b90f4a with peer a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:31.292507 12953 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:31.292874 12953 tablet_bootstrap.cc:492] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: Bootstrap complete.
I20240503 14:15:31.293128 12953 ts_tablet_manager.cc:1367] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20240503 14:15:31.293844 12953 raft_consensus.cc:348] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:31.294039 12953 raft_consensus.cc:729] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Initialized, Role: FOLLOWER
I20240503 14:15:31.295436 12953 consensus_queue.cc:260] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:31.296044 12953 ts_tablet_manager.cc:1398] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:15:31.298552 28194 tablet_copy_service.cc:342] P 6409961ee8c0445687c11724213c44a9: Request end of tablet copy session 530893a961af4b95b6b687fa5ed22f9d-a82470f287274b1c878089fa9d56ef40 received from {username='slave'} at 127.20.240.72:59619
I20240503 14:15:31.298672 28194 tablet_copy_service.cc:433] P 6409961ee8c0445687c11724213c44a9: ending tablet copy session 530893a961af4b95b6b687fa5ed22f9d-a82470f287274b1c878089fa9d56ef40 on tablet a82470f287274b1c878089fa9d56ef40 with peer 530893a961af4b95b6b687fa5ed22f9d
I20240503 14:15:31.416707 31201 raft_consensus.cc:1203] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:31.448323 28433 raft_consensus.cc:1203] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:31.608332 28300 raft_consensus.cc:1203] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:31.760018 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:31.763438 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:31.766732 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:31.771152 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:31.775511 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:31.904774 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:31.923975 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:31.951745 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:32.061728 28172 consensus_queue.cc:237] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:32.063644 28434 raft_consensus.cc:1261] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:32.063647 30941 raft_consensus.cc:1261] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:32.064159 12956 consensus_queue.cc:1035] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:32.064499 12866 consensus_queue.cc:1035] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:32.067319 12866 raft_consensus.cc:2932] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:32.067960 30941 raft_consensus.cc:2932] T 02559d1ab4824298ba76a44a7f05303b P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:32.068486 28434 raft_consensus.cc:2932] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:32.077477 27902 catalog_manager.cc:5484] T 02559d1ab4824298ba76a44a7f05303b P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72) evicted. New cstate: current_term: 1 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:32.102389 31183 tablet_service.cc:1514] Processing DeleteTablet for tablet 02559d1ab4824298ba76a44a7f05303b with delete_type TABLET_DATA_TOMBSTONED (TS 530893a961af4b95b6b687fa5ed22f9d not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:38660
I20240503 14:15:32.118597 13073 tablet_replica.cc:330] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: stopping tablet replica
I20240503 14:15:32.119012 13073 raft_consensus.cc:2229] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:32.124056 13073 raft_consensus.cc:2258] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:32.124819 13073 ts_tablet_manager.cc:1875] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:32.129985 13073 ts_tablet_manager.cc:1888] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:32.130069 13073 log.cc:1196] T 02559d1ab4824298ba76a44a7f05303b P 530893a961af4b95b6b687fa5ed22f9d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/wals/02559d1ab4824298ba76a44a7f05303b
I20240503 14:15:32.130875 27887 catalog_manager.cc:4830] TS 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331): tablet 02559d1ab4824298ba76a44a7f05303b (table rebalancer_extra_table_12 [id=91d24d2ac8f7438fa2967ca61fb59b9d]) successfully deleted
I20240503 14:15:32.380290 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:32.381220 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:32.402027 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:32.406092 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:32.406994 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:32.458145 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:32.470495 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:32.480787 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:32.720182 31069 consensus_queue.cc:237] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:32.722043 31201 raft_consensus.cc:1261] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:32.722064 28302 raft_consensus.cc:1261] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:32.722591 12849 consensus_queue.cc:1035] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:32.722829 11922 consensus_queue.cc:1035] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:32.738133 12849 raft_consensus.cc:2932] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:32.742022 28302 raft_consensus.cc:2932] T 917de496d0cb4c6d94c6dcf630b90f4a P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:32.742655 31201 raft_consensus.cc:2932] T 917de496d0cb4c6d94c6dcf630b90f4a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:32.746047 27898 catalog_manager.cc:5484] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index 2 to 3, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) evicted. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:32.767302 30789 tablet_service.cc:1514] Processing DeleteTablet for tablet 917de496d0cb4c6d94c6dcf630b90f4a with delete_type TABLET_DATA_TOMBSTONED (TS d0dbfb0e43f948c3922926aacc365fc1 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:48954
I20240503 14:15:32.770576 13152 tablet_replica.cc:330] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: stopping tablet replica
I20240503 14:15:32.770789 13152 raft_consensus.cc:2229] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:32.770988 13152 raft_consensus.cc:2258] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:32.771528 13152 ts_tablet_manager.cc:1875] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:32.776327 13152 ts_tablet_manager.cc:1888] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:32.776535 13152 log.cc:1196] T 917de496d0cb4c6d94c6dcf630b90f4a P d0dbfb0e43f948c3922926aacc365fc1: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/wals/917de496d0cb4c6d94c6dcf630b90f4a
I20240503 14:15:32.777197 27886 catalog_manager.cc:4830] TS d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): tablet 917de496d0cb4c6d94c6dcf630b90f4a (table rebalancer_extra_table_6 [id=06682f622ac341c293100bd740ae19cc]) successfully deleted
I20240503 14:15:33.028810 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:33.037834 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:33.050127 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:33.073285 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:33.083308 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:33.089651 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:33.109613 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:33.110410 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240503 14:15:33.235293 30745 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 57851.3us
I20240503 14:15:33.306676 30940 consensus_queue.cc:237] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:33.308676 31199 raft_consensus.cc:1261] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:33.309384 28433 raft_consensus.cc:1261] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:15:33.309479 13125 consensus_queue.cc:1035] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:33.310196 13125 consensus_queue.cc:1035] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:15:33.311815 12388 raft_consensus.cc:2932] T 1b4544aefcc54a828dc3b75726367f70 P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:33.313158 28433 raft_consensus.cc:2932] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:33.315999 31201 raft_consensus.cc:2932] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:33.320535 27895 catalog_manager.cc:5484] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 reported cstate change: config changed from index 2 to 3, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) evicted. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:33.327153 28283 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b4544aefcc54a828dc3b75726367f70 with delete_type TABLET_DATA_TOMBSTONED (TS a4cdf05256a0407c8929ac43eb55d44f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43626
I20240503 14:15:33.334882 13234 tablet_replica.cc:330] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: stopping tablet replica
I20240503 14:15:33.335094 13234 raft_consensus.cc:2229] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:33.336498 13234 raft_consensus.cc:2258] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:33.337103 13234 ts_tablet_manager.cc:1875] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:33.343513 13234 ts_tablet_manager.cc:1888] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240503 14:15:33.343611 13234 log.cc:1196] T 1b4544aefcc54a828dc3b75726367f70 P a4cdf05256a0407c8929ac43eb55d44f: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/wals/1b4544aefcc54a828dc3b75726367f70
I20240503 14:15:33.344368 27889 catalog_manager.cc:4830] TS a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): tablet 1b4544aefcc54a828dc3b75726367f70 (table rebalancer_extra_table_20 [id=445efe5ff037429fbd74555403a33d04]) successfully deleted
I20240503 14:15:33.861493 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:33.904541 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:33.913657 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:33.918334 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:33.996769 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:34.019759 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:34.046666 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:34.162968 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:34.294570 31199 consensus_queue.cc:237] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:34.296525 28432 raft_consensus.cc:1261] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:34.296516 28172 raft_consensus.cc:1261] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:34.297091 12323 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:34.297384 13287 consensus_queue.cc:1035] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:34.299886 12323 raft_consensus.cc:2932] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:34.302156 28172 raft_consensus.cc:2932] T 1af2fb3fb3fd4dfe873d124232d812b5 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:34.306174 27900 catalog_manager.cc:5484] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New cstate: current_term: 2 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:34.313025 28432 raft_consensus.cc:2932] T 1af2fb3fb3fd4dfe873d124232d812b5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:34.323195 31050 tablet_service.cc:1514] Processing DeleteTablet for tablet 1af2fb3fb3fd4dfe873d124232d812b5 with delete_type TABLET_DATA_TOMBSTONED (TS fde134d0fa2b4900b9badd2e08fea09d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45512
I20240503 14:15:34.323853 13365 tablet_replica.cc:330] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: stopping tablet replica
I20240503 14:15:34.324149 13365 raft_consensus.cc:2229] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:34.324441 13365 raft_consensus.cc:2258] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:34.325235 13365 ts_tablet_manager.cc:1875] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:34.331969 13365 ts_tablet_manager.cc:1888] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 14:15:34.332123 13365 log.cc:1196] T 1af2fb3fb3fd4dfe873d124232d812b5 P fde134d0fa2b4900b9badd2e08fea09d: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/wals/1af2fb3fb3fd4dfe873d124232d812b5
I20240503 14:15:34.332877 27888 catalog_manager.cc:4830] TS fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): tablet 1af2fb3fb3fd4dfe873d124232d812b5 (table rebalancer_extra_table_26 [id=6e198e465d8e4dd7b11d50595e3351fa]) successfully deleted
I20240503 14:15:34.741565 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:34.746858 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:34.764794 30921 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:34.824306 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:34.832572 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:34.836591 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:34.841548 28283 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:34.887434 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:35.034734 30941 consensus_queue.cc:237] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:35.037042 28300 raft_consensus.cc:1261] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:35.037022 28172 raft_consensus.cc:1261] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:35.037542 13350 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:35.037843 13230 consensus_queue.cc:1035] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:35.040614 13453 raft_consensus.cc:2932] T 05a67063845044eea2d56c34cdfd4ab1 P 6bbc686598f24f518d7ca6bb2063058d [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.041013 28172 raft_consensus.cc:2932] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.041126 28300 raft_consensus.cc:2932] T 05a67063845044eea2d56c34cdfd4ab1 P a4cdf05256a0407c8929ac43eb55d44f [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.047582 27900 catalog_manager.cc:5484] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 reported cstate change: config changed from index 4 to 6, VOTER 5388902973134958a030fd63ab4faba4 (127.20.240.68) evicted. New cstate: current_term: 2 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.057080 28412 tablet_service.cc:1514] Processing DeleteTablet for tablet 05a67063845044eea2d56c34cdfd4ab1 with delete_type TABLET_DATA_TOMBSTONED (TS 5388902973134958a030fd63ab4faba4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58086
I20240503 14:15:35.057706 13461 tablet_replica.cc:330] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: stopping tablet replica
I20240503 14:15:35.057921 13461 raft_consensus.cc:2229] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:15:35.058109 13461 raft_consensus.cc:2258] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:15:35.059435 13461 ts_tablet_manager.cc:1875] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:15:35.067768 13461 ts_tablet_manager.cc:1888] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240503 14:15:35.067894 13461 log.cc:1196] T 05a67063845044eea2d56c34cdfd4ab1 P 5388902973134958a030fd63ab4faba4: Deleting WAL directory at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/wals/05a67063845044eea2d56c34cdfd4ab1
I20240503 14:15:35.068686 27889 catalog_manager.cc:4830] TS 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): tablet 05a67063845044eea2d56c34cdfd4ab1 (table rebalance_test_table_9 [id=4673aaf684274f249d74ffa7649ceb73]) successfully deleted
I20240503 14:15:35.123428 31070 consensus_queue.cc:237] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:35.125795 30940 raft_consensus.cc:1261] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.125847 28040 raft_consensus.cc:1261] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.126331 13426 consensus_queue.cc:1035] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:35.126605 31006 consensus_peers.cc:485] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a88d8b316d064c4e8fdccaba77d656c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:35.126796 13426 consensus_queue.cc:1035] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.129076 13426 raft_consensus.cc:2932] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.131968 28040 raft_consensus.cc:2932] T a88d8b316d064c4e8fdccaba77d656c2 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.135422 30940 raft_consensus.cc:2932] T a88d8b316d064c4e8fdccaba77d656c2 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.138022 27896 catalog_manager.cc:5484] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.170089 31201 consensus_queue.cc:237] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:35.172624 30941 raft_consensus.cc:1261] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.172823 30810 raft_consensus.cc:1261] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.173314 12323 consensus_queue.cc:1035] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.173585 13287 consensus_queue.cc:1035] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:35.174142 31139 consensus_peers.cc:485] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f807f1ac8d8417aaffd9254993b2085. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:35.177572 13287 raft_consensus.cc:2932] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:35.178082 30941 raft_consensus.cc:2932] T 7f807f1ac8d8417aaffd9254993b2085 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:35.182168 30810 raft_consensus.cc:2932] T 7f807f1ac8d8417aaffd9254993b2085 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:35.190318 27899 catalog_manager.cc:5484] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) added. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:35.232636 31070 consensus_queue.cc:237] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:35.234763 28300 raft_consensus.cc:1261] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.234901 30810 raft_consensus.cc:1261] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:35.234951 31008 consensus_peers.cc:485] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b6b6e0cac9a4ea6a0189970d88f9d72. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:35.235332 13426 consensus_queue.cc:1035] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.235606 12849 consensus_queue.cc:1035] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.238945 12849 raft_consensus.cc:2932] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:35.239537 28300 raft_consensus.cc:2932] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:35.246776 27896 catalog_manager.cc:5484] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P a4cdf05256a0407c8929ac43eb55d44f reported cstate change: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New cstate: current_term: 1 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:35.247624 30810 raft_consensus.cc:2932] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:35.307379 30940 consensus_queue.cc:237] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:35.309589 28041 raft_consensus.cc:1261] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.309588 31201 raft_consensus.cc:1261] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer 6bbc686598f24f518d7ca6bb2063058d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.310140 13453 consensus_queue.cc:1035] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.310395 13230 consensus_queue.cc:1035] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:35.310894 30875 consensus_peers.cc:485] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 361b2d07d8b44d128acfb3cc941ac01a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:35.313887 13230 raft_consensus.cc:2932] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.314554 28041 raft_consensus.cc:2932] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.319288 31199 raft_consensus.cc:2932] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.320802 27896 catalog_manager.cc:5484] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: config changed from index -1 to 2, VOTER 6409961ee8c0445687c11724213c44a9 (127.20.240.66) added. New cstate: current_term: 1 leader_uuid: "6bbc686598f24f518d7ca6bb2063058d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:35.367008 31199 consensus_queue.cc:237] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:35.369215 30941 raft_consensus.cc:1261] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:35.369356 31136 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 133f5691d69b461f8e7502ecffdb8a89. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:35.369388 28300 raft_consensus.cc:1261] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.369827 13287 consensus_queue.cc:1035] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.370158 13503 consensus_queue.cc:1035] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.379206 13287 raft_consensus.cc:2932] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:35.382637 30941 raft_consensus.cc:2932] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:35.381860 28300 raft_consensus.cc:2932] T 133f5691d69b461f8e7502ecffdb8a89 P a4cdf05256a0407c8929ac43eb55d44f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:35.395123 27899 catalog_manager.cc:5484] T 133f5691d69b461f8e7502ecffdb8a89 P 6bbc686598f24f518d7ca6bb2063058d reported cstate change: config changed from index -1 to 2, VOTER d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69) added. New cstate: current_term: 1 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:35.444602 30810 consensus_queue.cc:237] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
W20240503 14:15:35.447502 30745 consensus_peers.cc:485] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 027c88573444465a9ebcaa9589728e74. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:35.448107 31199 raft_consensus.cc:1261] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.448745 31070 raft_consensus.cc:1261] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.448755 13504 consensus_queue.cc:1035] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.449433 13504 consensus_queue.cc:1035] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.451941 13210 raft_consensus.cc:2932] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.452675 31070 raft_consensus.cc:2932] T 027c88573444465a9ebcaa9589728e74 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.456692 31199 raft_consensus.cc:2932] T 027c88573444465a9ebcaa9589728e74 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.459797 27899 catalog_manager.cc:5484] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.494134 30810 consensus_queue.cc:237] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:35.497056 31070 raft_consensus.cc:1261] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:35.497085 31199 raft_consensus.cc:1261] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:15:35.497167 30745 consensus_peers.cc:485] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bca56ccf82a74df7bd8006e31e02bd20. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:35.497606 13504 consensus_queue.cc:1035] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.498092 12392 consensus_queue.cc:1035] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:35.506898 12392 raft_consensus.cc:2932] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.511078 31199 raft_consensus.cc:2932] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.520113 27899 catalog_manager.cc:5484] T bca56ccf82a74df7bd8006e31e02bd20 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New cstate: current_term: 1 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.531013 31070 raft_consensus.cc:2932] T bca56ccf82a74df7bd8006e31e02bd20 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } }
I20240503 14:15:35.567749 13610 ts_tablet_manager.cc:902] T 7f807f1ac8d8417aaffd9254993b2085 P a4cdf05256a0407c8929ac43eb55d44f: Initiating tablet copy from peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
F20240503 14:15:35.568100 13610 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-029be0389e244435bb08db2361729ad2.queue_time_us, tablet-download-pool-029be0389e244435bb08db2361729ad2.queue_time_us) (tablet-download-pool-029be0389e244435bb08db2361729ad2.run_cpu_time_us, tablet-download-pool-029be0389e244435bb08db2361729ad2.run_cpu_time_us) (tablet-download-pool-029be0389e244435bb08db2361729ad2.run_wall_time_us, tablet-download-pool-029be0389e244435bb08db2361729ad2.run_wall_time_us) (tablet-download-pool-06e418c4c2cf46608ba454d48fae9285.queue_time_us, tablet-download-pool-06e418c4c2cf46608ba454d48fae9285.queue_time_us) (tablet-download-pool-06e418c4c2cf46608ba454d48fae9285.run_cpu_time_us, tablet-download-pool-06e418c4c2cf46608ba454d48fae9285.run_cpu_time_us) (tablet-download-pool-06e418c4c2cf46608ba454d48fae9285.run_wall_time_us, tablet-download-pool-06e418c4c2cf46608ba454d48fae9285.run_wall_time_us) (tablet-download-pool-2d0b457ba3564c52824f84b78cf14c23.queue_time_us, tablet-download-pool-2d0b457ba3564c52824f84b78cf14c23.queue_time_us) (tablet-download-pool-2d0b457ba3564c52824f84b78cf14c23.run_cpu_time_us, tablet-download-pool-2d0b457ba3564c52824f84b78cf14c23.run_cpu_time_us) (tablet-download-pool-2d0b457ba3564c52824f84b78cf14c23.run_wall_time_us, tablet-download-pool-2d0b457ba3564c52824f84b78cf14c23.run_wall_time_us) (tablet-download-pool-5be4220923ed4c5388a7a15755cb685d.queue_time_us, tablet-download-pool-5be4220923ed4c5388a7a15755cb685d.queue_time_us) (tablet-download-pool-5be4220923ed4c5388a7a15755cb685d.run_cpu_time_us, tablet-download-pool-5be4220923ed4c5388a7a15755cb685d.run_cpu_time_us) (tablet-download-pool-5be4220923ed4c5388a7a15755cb685d.run_wall_time_us, tablet-download-pool-5be4220923ed4c5388a7a15755cb685d.run_wall_time_us) (tablet-download-pool-619951b364d44766a2f3064222ceb738.queue_time_us, tablet-download-pool-619951b364d44766a2f3064222ceb738.queue_time_us) (tablet-download-pool-619951b364d44766a2f3064222ceb738.run_cpu_time_us, tablet-download-pool-619951b364d44766a2f3064222ceb738.run_cpu_time_us) (tablet-download-pool-619951b364d44766a2f3064222ceb738.run_wall_time_us, tablet-download-pool-619951b364d44766a2f3064222ceb738.run_wall_time_us) (tablet-download-pool-7f807f1ac8d8417aaffd9254993b2085.queue_time_us, tablet-download-pool-7f807f1ac8d8417aaffd9254993b2085.queue_time_us) (tablet-download-pool-917de496d0cb4c6d94c6dcf630b90f4a.queue_time_us, tablet-download-pool-917de496d0cb4c6d94c6dcf630b90f4a.queue_time_us) (tablet-download-pool-917de496d0cb4c6d94c6dcf630b90f4a.run_cpu_time_us, tablet-download-pool-917de496d0cb4c6d94c6dcf630b90f4a.run_cpu_time_us) (tablet-download-pool-917de496d0cb4c6d94c6dcf630b90f4a.run_wall_time_us, tablet-download-pool-917de496d0cb4c6d94c6dcf630b90f4a.run_wall_time_us) (tablet-download-pool-96773dd7242d4223bdc663ee53bfdd3f.queue_time_us, tablet-download-pool-96773dd7242d4223bdc663ee53bfdd3f.queue_time_us) (tablet-download-pool-96773dd7242d4223bdc663ee53bfdd3f.run_cpu_time_us, tablet-download-pool-96773dd7242d4223bdc663ee53bfdd3f.run_cpu_time_us) (tablet-download-pool-96773dd7242d4223bdc663ee53bfdd3f.run_wall_time_us, tablet-download-pool-96773dd7242d4223bdc663ee53bfdd3f.run_wall_time_us) (tablet-download-pool-a3888675106d4d389718380627c95edf.queue_time_us, tablet-download-pool-a3888675106d4d389718380627c95edf.queue_time_us) (tablet-download-pool-a3888675106d4d389718380627c95edf.run_cpu_time_us, tablet-download-pool-a3888675106d4d389718380627c95edf.run_cpu_time_us) (tablet-download-pool-a3888675106d4d389718380627c95edf.run_wall_time_us, tablet-download-pool-a3888675106d4d389718380627c95edf.run_wall_time_us) (tablet-download-pool-bb1ec320c4114eec8915b7f2bcffc2c3.queue_time_us, tablet-download-pool-bb1ec320c4114eec8915b7f2bcffc2c3.queue_time_us) (tablet-download-pool-bb1ec320c4114eec8915b7f2bcffc2c3.run_cpu_time_us, tablet-download-pool-bb1ec320c4114eec8915b7f2bcffc2c3.run_cpu_time_us) (tablet-download-pool-bb1ec320c4114eec8915b7f2bcffc2c3.run_wall_time_us, tablet-download-pool-bb1ec320c4114eec8915b7f2bcffc2c3.run_wall_time_us) (tablet-download-pool-d58ff895fc8e4ddb84e2c1db165799d1.queue_time_us, tablet-download-pool-d58ff895fc8e4ddb84e2c1db165799d1.queue_time_us) (tablet-download-pool-d58ff895fc8e4ddb84e2c1db165799d1.run_cpu_time_us, tablet-download-pool-d58ff895fc8e4ddb84e2c1db165799d1.run_cpu_time_us) (tablet-download-pool-d58ff895fc8e4ddb84e2c1db165799d1.run_wall_time_us, tablet-download-pool-d58ff895fc8e4ddb84e2c1db165799d1.run_wall_time_us) (tablet-download-pool-dcb14382755441afb4cc5c103425331c.queue_time_us, tablet-download-pool-dcb14382755441afb4cc5c103425331c.queue_time_us) (tablet-download-pool-dcb14382755441afb4cc5c103425331c.run_cpu_time_us, tablet-download-pool-dcb14382755441afb4cc5c103425331c.run_cpu_time_us) (tablet-download-pool-dcb14382755441afb4cc5c103425331c.run_wall_time_us, tablet-download-pool-dcb14382755441afb4cc5c103425331c.run_wall_time_us) (tablet-download-pool-e389fcad67cc4add9db2f34eb57745c8.queue_time_us, tablet-download-pool-e389fcad67cc4add9db2f34eb57745c8.queue_time_us) (tablet-download-pool-e389fcad67cc4add9db2f34eb57745c8.run_cpu_time_us, tablet-download-pool-e389fcad67cc4add9db2f34eb57745c8.run_cpu_time_us) (tablet-download-pool-e389fcad67cc4add9db2f34eb57745c8.run_wall_time_us, tablet-download-pool-e389fcad67cc4add9db2f34eb57745c8.run_wall_time_us) (tablet-download-pool-e571695c819e485483ece09278ee1e09.queue_time_us, tablet-download-pool-e571695c819e485483ece09278ee1e09.queue_time_us) (tablet-download-pool-e571695c819e485483ece09278ee1e09.run_cpu_time_us, tablet-download-pool-e571695c819e485483ece09278ee1e09.run_cpu_time_us) (tablet-download-pool-e571695c819e485483ece09278ee1e09.run_wall_time_us, tablet-download-pool-e571695c819e485483ece09278ee1e09.run_wall_time_us) (tablet-download-pool-e57c283674c342169940a5903516ec93.queue_time_us, tablet-download-pool-e57c283674c342169940a5903516ec93.queue_time_us) (tablet-download-pool-e57c283674c342169940a5903516ec93.run_cpu_time_us, tablet-download-pool-e57c283674c342169940a5903516ec93.run_cpu_time_us) (tablet-download-pool-e57c283674c342169940a5903516ec93.run_wall_time_us, tablet-download-pool-e57c283674c342169940a5903516ec93.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f5f6068ddcd google::LogMessage::Fail() at ??:0
@ 0x7f5f60691b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f5f6068d7cc google::LogMessage::Flush() at ??:0
@ 0x7f5f6068ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5f633ce742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f5f6339af8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f5f63394cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f5f727e88de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f5f72800daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f5f7292b480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f5f72963a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f5f72946981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f5f7294670e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5f6339eb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5f633a219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f5f633a2019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5f6337d01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5f667026db start_thread at ??:0
@ 0x7f5f5f54071f clone at ??:0
I20240503 14:15:35.743613 13635 ts_tablet_manager.cc:902] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:35.789793 13635 tablet_copy_client.cc:308] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:15:35.790596 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet a88d8b316d064c4e8fdccaba77d656c2 from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:42465)
I20240503 14:15:35.790760 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet a88d8b316d064c4e8fdccaba77d656c2 from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:42465: session id = 6409961ee8c0445687c11724213c44a9-a88d8b316d064c4e8fdccaba77d656c2
I20240503 14:15:35.793401 31092 tablet_copy_source_session.cc:205] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:35.794888 13635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a88d8b316d064c4e8fdccaba77d656c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:35.798525 13644 ts_tablet_manager.cc:902] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: Initiating tablet copy from peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:35.808302 13635 tablet_copy_client.cc:783] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:35.809330 13635 tablet_copy_client.cc:647] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:35.811502 13635 tablet_copy_client.cc:523] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:35.822889 13635 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:15:35.834167 13644 tablet_copy_client.cc:308] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.71:34707
I20240503 14:15:35.836021 31092 tablet_copy_service.cc:140] P fde134d0fa2b4900b9badd2e08fea09d: Received BeginTabletCopySession request for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72 from peer 793a0d5dd1634c5faaf0e1e4851e8294 ({username='slave'} at 127.20.240.65:43505)
I20240503 14:15:35.836141 31092 tablet_copy_service.cc:161] P fde134d0fa2b4900b9badd2e08fea09d: Beginning new tablet copy session on tablet 4b6b6e0cac9a4ea6a0189970d88f9d72 from peer 793a0d5dd1634c5faaf0e1e4851e8294 at {username='slave'} at 127.20.240.65:43505: session id = 793a0d5dd1634c5faaf0e1e4851e8294-4b6b6e0cac9a4ea6a0189970d88f9d72
I20240503 14:15:35.838735 31092 tablet_copy_source_session.cc:205] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:35.841183 13661 ts_tablet_manager.cc:902] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:35.842545 13644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b6b6e0cac9a4ea6a0189970d88f9d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:35.845894 13635 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:35.846369 13635 tablet_bootstrap.cc:492] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:15:35.846621 13635 ts_tablet_manager.cc:1367] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.007s
I20240503 14:15:35.850344 13635 raft_consensus.cc:348] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:35.850610 13635 raft_consensus.cc:729] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:15:35.850909 13635 consensus_queue.cc:260] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:35.852234 13644 tablet_copy_client.cc:783] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:35.852427 13644 tablet_copy_client.cc:647] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:35.854861 13635 ts_tablet_manager.cc:1398] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:15:35.855566 31091 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-a88d8b316d064c4e8fdccaba77d656c2 received from {username='slave'} at 127.20.240.66:42465
I20240503 14:15:35.855660 31091 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session 6409961ee8c0445687c11724213c44a9-a88d8b316d064c4e8fdccaba77d656c2 on tablet a88d8b316d064c4e8fdccaba77d656c2 with peer 6409961ee8c0445687c11724213c44a9
I20240503 14:15:35.856469 13644 tablet_copy_client.cc:523] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:35.860652 13644 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap starting.
I20240503 14:15:35.891319 13661 tablet_copy_client.cc:308] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.69:39689
I20240503 14:15:35.892024 30830 tablet_copy_service.cc:140] P d0dbfb0e43f948c3922926aacc365fc1: Received BeginTabletCopySession request for tablet 027c88573444465a9ebcaa9589728e74 from peer 6bbc686598f24f518d7ca6bb2063058d ({username='slave'} at 127.20.240.70:45791)
I20240503 14:15:35.892164 30830 tablet_copy_service.cc:161] P d0dbfb0e43f948c3922926aacc365fc1: Beginning new tablet copy session on tablet 027c88573444465a9ebcaa9589728e74 from peer 6bbc686598f24f518d7ca6bb2063058d at {username='slave'} at 127.20.240.70:45791: session id = 6bbc686598f24f518d7ca6bb2063058d-027c88573444465a9ebcaa9589728e74
I20240503 14:15:35.895246 30830 tablet_copy_source_session.cc:205] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:35.898293 13661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 027c88573444465a9ebcaa9589728e74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:35.904847 13661 tablet_copy_client.cc:783] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:35.905076 13661 tablet_copy_client.cc:647] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:35.906805 13661 tablet_copy_client.cc:523] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:35.910388 13661 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap starting.
I20240503 14:15:35.921030 13644 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:35.921458 13644 tablet_bootstrap.cc:492] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: Bootstrap complete.
I20240503 14:15:35.921698 13644 ts_tablet_manager.cc:1367] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.012s
I20240503 14:15:35.922494 13644 raft_consensus.cc:348] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:35.922703 13644 raft_consensus.cc:729] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Initialized, Role: FOLLOWER
I20240503 14:15:35.922945 13644 consensus_queue.cc:260] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:35.924013 13644 ts_tablet_manager.cc:1398] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 14:15:35.924867 31091 tablet_copy_service.cc:342] P fde134d0fa2b4900b9badd2e08fea09d: Request end of tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-4b6b6e0cac9a4ea6a0189970d88f9d72 received from {username='slave'} at 127.20.240.65:43505
I20240503 14:15:35.925019 31091 tablet_copy_service.cc:433] P fde134d0fa2b4900b9badd2e08fea09d: ending tablet copy session 793a0d5dd1634c5faaf0e1e4851e8294-4b6b6e0cac9a4ea6a0189970d88f9d72 on tablet 4b6b6e0cac9a4ea6a0189970d88f9d72 with peer 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:35.931658 13635 ts_tablet_manager.cc:902] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: Initiating tablet copy from peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:35.932981 13661 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:35.933338 13661 tablet_bootstrap.cc:492] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: Bootstrap complete.
I20240503 14:15:35.933584 13661 ts_tablet_manager.cc:1367] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.008s
I20240503 14:15:35.934417 13661 raft_consensus.cc:348] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:35.936169 13661 raft_consensus.cc:729] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bbc686598f24f518d7ca6bb2063058d, State: Initialized, Role: FOLLOWER
I20240503 14:15:35.936712 13661 consensus_queue.cc:260] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:35.938956 13661 ts_tablet_manager.cc:1398] T 027c88573444465a9ebcaa9589728e74 P 6bbc686598f24f518d7ca6bb2063058d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:15:35.939574 30830 tablet_copy_service.cc:342] P d0dbfb0e43f948c3922926aacc365fc1: Request end of tablet copy session 6bbc686598f24f518d7ca6bb2063058d-027c88573444465a9ebcaa9589728e74 received from {username='slave'} at 127.20.240.70:45791
I20240503 14:15:35.939697 30830 tablet_copy_service.cc:433] P d0dbfb0e43f948c3922926aacc365fc1: ending tablet copy session 6bbc686598f24f518d7ca6bb2063058d-027c88573444465a9ebcaa9589728e74 on tablet 027c88573444465a9ebcaa9589728e74 with peer 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:35.955859 13635 tablet_copy_client.cc:308] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.70:39465
I20240503 14:15:35.956532 30961 tablet_copy_service.cc:140] P 6bbc686598f24f518d7ca6bb2063058d: Received BeginTabletCopySession request for tablet 361b2d07d8b44d128acfb3cc941ac01a from peer 6409961ee8c0445687c11724213c44a9 ({username='slave'} at 127.20.240.66:54479)
I20240503 14:15:35.956669 30961 tablet_copy_service.cc:161] P 6bbc686598f24f518d7ca6bb2063058d: Beginning new tablet copy session on tablet 361b2d07d8b44d128acfb3cc941ac01a from peer 6409961ee8c0445687c11724213c44a9 at {username='slave'} at 127.20.240.66:54479: session id = 6409961ee8c0445687c11724213c44a9-361b2d07d8b44d128acfb3cc941ac01a
I20240503 14:15:35.959767 30961 tablet_copy_source_session.cc:205] T 361b2d07d8b44d128acfb3cc941ac01a P 6bbc686598f24f518d7ca6bb2063058d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:35.961289 13635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361b2d07d8b44d128acfb3cc941ac01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:35.961849 13680 ts_tablet_manager.cc:902] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: Initiating tablet copy from peer 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:35.974067 13635 tablet_copy_client.cc:783] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:35.974315 13635 tablet_copy_client.cc:647] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:35.976697 13635 tablet_copy_client.cc:523] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:35.983829 13635 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: Bootstrap starting.
I20240503 14:15:36.006168 13680 tablet_copy_client.cc:308] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.20.240.72:41331
I20240503 14:15:36.007074 31223 tablet_copy_service.cc:140] P 530893a961af4b95b6b687fa5ed22f9d: Received BeginTabletCopySession request for tablet 133f5691d69b461f8e7502ecffdb8a89 from peer d0dbfb0e43f948c3922926aacc365fc1 ({username='slave'} at 127.20.240.69:35455)
I20240503 14:15:36.007218 31223 tablet_copy_service.cc:161] P 530893a961af4b95b6b687fa5ed22f9d: Beginning new tablet copy session on tablet 133f5691d69b461f8e7502ecffdb8a89 from peer d0dbfb0e43f948c3922926aacc365fc1 at {username='slave'} at 127.20.240.69:35455: session id = d0dbfb0e43f948c3922926aacc365fc1-133f5691d69b461f8e7502ecffdb8a89
I20240503 14:15:36.009413 31223 tablet_copy_source_session.cc:205] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:15:36.015622 13680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 133f5691d69b461f8e7502ecffdb8a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:36.025691 13680 tablet_copy_client.cc:783] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 0 data blocks...
I20240503 14:15:36.025930 13680 tablet_copy_client.cc:647] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Starting download of 1 WAL segments...
I20240503 14:15:36.028138 13680 tablet_copy_client.cc:523] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:15:36.030396 13635 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:15:36.030918 13635 tablet_bootstrap.cc:492] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: Bootstrap complete.
I20240503 14:15:36.032668 13680 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap starting.
I20240503 14:15:36.039886 13635 ts_tablet_manager.cc:1367] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.003s
I20240503 14:15:36.040658 13635 raft_consensus.cc:348] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:36.041013 13635 raft_consensus.cc:729] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.044309 13635 consensus_queue.cc:260] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:36.052529 13661 ts_tablet_manager.cc:902] T bca56ccf82a74df7bd8006e31e02bd20 P 6bbc686598f24f518d7ca6bb2063058d: Initiating tablet copy from peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:36.052886 13680 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
F20240503 14:15:36.052821 13661 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0225203f2de2498eb448e066ce545a64.queue_time_us, tablet-download-pool-0225203f2de2498eb448e066ce545a64.queue_time_us) (tablet-download-pool-0225203f2de2498eb448e066ce545a64.run_cpu_time_us, tablet-download-pool-0225203f2de2498eb448e066ce545a64.run_cpu_time_us) (tablet-download-pool-0225203f2de2498eb448e066ce545a64.run_wall_time_us, tablet-download-pool-0225203f2de2498eb448e066ce545a64.run_wall_time_us) (tablet-download-pool-027c88573444465a9ebcaa9589728e74.queue_time_us, tablet-download-pool-027c88573444465a9ebcaa9589728e74.queue_time_us) (tablet-download-pool-027c88573444465a9ebcaa9589728e74.run_cpu_time_us, tablet-download-pool-027c88573444465a9ebcaa9589728e74.run_cpu_time_us) (tablet-download-pool-027c88573444465a9ebcaa9589728e74.run_wall_time_us, tablet-download-pool-027c88573444465a9ebcaa9589728e74.run_wall_time_us) (tablet-download-pool-05a67063845044eea2d56c34cdfd4ab1.queue_time_us, tablet-download-pool-05a67063845044eea2d56c34cdfd4ab1.queue_time_us) (tablet-download-pool-05a67063845044eea2d56c34cdfd4ab1.run_cpu_time_us, tablet-download-pool-05a67063845044eea2d56c34cdfd4ab1.run_cpu_time_us) (tablet-download-pool-05a67063845044eea2d56c34cdfd4ab1.run_wall_time_us, tablet-download-pool-05a67063845044eea2d56c34cdfd4ab1.run_wall_time_us) (tablet-download-pool-2e1fbc1d1bb6439c9f8b64364ad9ab4b.queue_time_us, tablet-download-pool-2e1fbc1d1bb6439c9f8b64364ad9ab4b.queue_time_us) (tablet-download-pool-2e1fbc1d1bb6439c9f8b64364ad9ab4b.run_cpu_time_us, tablet-download-pool-2e1fbc1d1bb6439c9f8b64364ad9ab4b.run_cpu_time_us) (tablet-download-pool-2e1fbc1d1bb6439c9f8b64364ad9ab4b.run_wall_time_us, tablet-download-pool-2e1fbc1d1bb6439c9f8b64364ad9ab4b.run_wall_time_us) (tablet-download-pool-344eb942dc3d4188bc9baf3d1ecc4f7a.queue_time_us, tablet-download-pool-344eb942dc3d4188bc9baf3d1ecc4f7a.queue_time_us) (tablet-download-pool-344eb942dc3d4188bc9baf3d1ecc4f7a.run_cpu_time_us, tablet-download-pool-344eb942dc3d4188bc9baf3d1ecc4f7a.run_cpu_time_us) (tablet-download-pool-344eb942dc3d4188bc9baf3d1ecc4f7a.run_wall_time_us, tablet-download-pool-344eb942dc3d4188bc9baf3d1ecc4f7a.run_wall_time_us) (tablet-download-pool-38cf3aab9e1b4f28842ac16d02be2562.queue_time_us, tablet-download-pool-38cf3aab9e1b4f28842ac16d02be2562.queue_time_us) (tablet-download-pool-38cf3aab9e1b4f28842ac16d02be2562.run_cpu_time_us, tablet-download-pool-38cf3aab9e1b4f28842ac16d02be2562.run_cpu_time_us) (tablet-download-pool-38cf3aab9e1b4f28842ac16d02be2562.run_wall_time_us, tablet-download-pool-38cf3aab9e1b4f28842ac16d02be2562.run_wall_time_us) (tablet-download-pool-45da3dab7a204d378273a8985d91ace8.queue_time_us, tablet-download-pool-45da3dab7a204d378273a8985d91ace8.queue_time_us) (tablet-download-pool-45da3dab7a204d378273a8985d91ace8.run_cpu_time_us, tablet-download-pool-45da3dab7a204d378273a8985d91ace8.run_cpu_time_us) (tablet-download-pool-45da3dab7a204d378273a8985d91ace8.run_wall_time_us, tablet-download-pool-45da3dab7a204d378273a8985d91ace8.run_wall_time_us) (tablet-download-pool-49d89642d5364479877c7ba4700fdcee.queue_time_us, tablet-download-pool-49d89642d5364479877c7ba4700fdcee.queue_time_us) (tablet-download-pool-49d89642d5364479877c7ba4700fdcee.run_cpu_time_us, tablet-download-pool-49d89642d5364479877c7ba4700fdcee.run_cpu_time_us) (tablet-download-pool-49d89642d5364479877c7ba4700fdcee.run_wall_time_us, tablet-download-pool-49d89642d5364479877c7ba4700fdcee.run_wall_time_us) (tablet-download-pool-519fed0fc03b4b9788803aeb0bbcfdad.queue_time_us, tablet-download-pool-519fed0fc03b4b9788803aeb0bbcfdad.queue_time_us) (tablet-download-pool-519fed0fc03b4b9788803aeb0bbcfdad.run_cpu_time_us, tablet-download-pool-519fed0fc03b4b9788803aeb0bbcfdad.run_cpu_time_us) (tablet-download-pool-519fed0fc03b4b9788803aeb0bbcfdad.run_wall_time_us, tablet-download-pool-519fed0fc03b4b9788803aeb0bbcfdad.run_wall_time_us) (tablet-download-pool-5f25a3bcdef849499e4a02f01f4dbc0e.queue_time_us, tablet-download-pool-5f25a3bcdef849499e4a02f01f4dbc0e.queue_time_us) (tablet-download-pool-5f25a3bcdef849499e4a02f01f4dbc0e.run_cpu_time_us, tablet-download-pool-5f25a3bcdef849499e4a02f01f4dbc0e.run_cpu_time_us) (tablet-download-pool-5f25a3bcdef849499e4a02f01f4dbc0e.run_wall_time_us, tablet-download-pool-5f25a3bcdef849499e4a02f01f4dbc0e.run_wall_time_us) (tablet-download-pool-bca56ccf82a74df7bd8006e31e02bd20.queue_time_us, tablet-download-pool-bca56ccf82a74df7bd8006e31e02bd20.queue_time_us) (tablet-download-pool-cb88c79b4b1a40c2b45c2dd79ab894d4.queue_time_us, tablet-download-pool-cb88c79b4b1a40c2b45c2dd79ab894d4.queue_time_us) (tablet-download-pool-cb88c79b4b1a40c2b45c2dd79ab894d4.run_cpu_time_us, tablet-download-pool-cb88c79b4b1a40c2b45c2dd79ab894d4.run_cpu_time_us) (tablet-download-pool-cb88c79b4b1a40c2b45c2dd79ab894d4.run_wall_time_us, tablet-download-pool-cb88c79b4b1a40c2b45c2dd79ab894d4.run_wall_time_us) (tablet-download-pool-dc9c29880eea4cd9a11eb24eeb3bd228.queue_time_us, tablet-download-pool-dc9c29880eea4cd9a11eb24eeb3bd228.queue_time_us) (tablet-download-pool-dc9c29880eea4cd9a11eb24eeb3bd228.run_cpu_time_us, tablet-download-pool-dc9c29880eea4cd9a11eb24eeb3bd228.run_cpu_time_us) (tablet-download-pool-dc9c29880eea4cd9a11eb24eeb3bd228.run_wall_time_us, tablet-download-pool-dc9c29880eea4cd9a11eb24eeb3bd228.run_wall_time_us) (tablet-download-pool-f133ebe695a74f86be1a37ae661719ec.queue_time_us, tablet-download-pool-f133ebe695a74f86be1a37ae661719ec.queue_time_us) (tablet-download-pool-f133ebe695a74f86be1a37ae661719ec.run_cpu_time_us, tablet-download-pool-f133ebe695a74f86be1a37ae661719ec.run_cpu_time_us) (tablet-download-pool-f133ebe695a74f86be1a37ae661719ec.run_wall_time_us, tablet-download-pool-f133ebe695a74f86be1a37ae661719ec.run_wall_time_us) (tablet-download-pool-f4b9d916f3fb4a348a8702662c143435.queue_time_us, tablet-download-pool-f4b9d916f3fb4a348a8702662c143435.queue_time_us) (tablet-download-pool-f4b9d916f3fb4a348a8702662c143435.run_cpu_time_us, tablet-download-pool-f4b9d916f3fb4a348a8702662c143435.run_cpu_time_us) (tablet-download-pool-f4b9d916f3fb4a348a8702662c143435.run_wall_time_us, tablet-download-pool-f4b9d916f3fb4a348a8702662c143435.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 14:15:36.053212 13680 tablet_bootstrap.cc:492] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: Bootstrap complete.
*** Check failure stack trace: ***
I20240503 14:15:36.053576 13680 ts_tablet_manager.cc:1367] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20240503 14:15:36.054563 13680 raft_consensus.cc:348] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:36.054831 13680 raft_consensus.cc:729] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Initialized, Role: FOLLOWER
I20240503 14:15:36.055104 13680 consensus_queue.cc:260] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
@ 0x7fa38a197dcd google::LogMessage::Fail() at ??:0
@ 0x7fa38a19bb93 google::LogMessage::SendToLog() at ??:0
I20240503 14:15:36.058903 13680 ts_tablet_manager.cc:1398] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:15:36.059581 31223 tablet_copy_service.cc:342] P 530893a961af4b95b6b687fa5ed22f9d: Request end of tablet copy session d0dbfb0e43f948c3922926aacc365fc1-133f5691d69b461f8e7502ecffdb8a89 received from {username='slave'} at 127.20.240.69:35455
@ 0x7fa38a1977cc google::LogMessage::Flush() at ??:0
I20240503 14:15:36.059727 31223 tablet_copy_service.cc:433] P 530893a961af4b95b6b687fa5ed22f9d: ending tablet copy session d0dbfb0e43f948c3922926aacc365fc1-133f5691d69b461f8e7502ecffdb8a89 on tablet 133f5691d69b461f8e7502ecffdb8a89 with peer d0dbfb0e43f948c3922926aacc365fc1
@ 0x7fa38a198f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 14:15:36.067993 13635 ts_tablet_manager.cc:1398] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 14:15:36.068895 30961 tablet_copy_service.cc:342] P 6bbc686598f24f518d7ca6bb2063058d: Request end of tablet copy session 6409961ee8c0445687c11724213c44a9-361b2d07d8b44d128acfb3cc941ac01a received from {username='slave'} at 127.20.240.66:54479
I20240503 14:15:36.069012 30961 tablet_copy_service.cc:433] P 6bbc686598f24f518d7ca6bb2063058d: ending tablet copy session 6409961ee8c0445687c11724213c44a9-361b2d07d8b44d128acfb3cc941ac01a on tablet 361b2d07d8b44d128acfb3cc941ac01a with peer 6409961ee8c0445687c11724213c44a9
@ 0x7fa38ced8742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fa38cea4f8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fa38ce9ecd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fa39c2f28de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
W20240503 14:15:36.099922 31008 connection.cc:535] client connection to 127.20.240.67:43889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.100212 31008 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7fa39c30adaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
W20240503 14:15:36.105113 30877 connection.cc:535] client connection to 127.20.240.67:43889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.105379 30877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.112589 30746 connection.cc:535] client connection to 127.20.240.67:43889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7fa39c435480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
W20240503 14:15:36.112839 30746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.123160 28370 connection.cc:535] client connection to 127.20.240.67:43889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.123217 28110 connection.cc:535] client connection to 127.20.240.67:43889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.123380 28370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.123507 28110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.123847 27979 connection.cc:535] client connection to 127.20.240.67:43889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.124135 27979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.125036 31139 connection.cc:535] client connection to 127.20.240.67:43889 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.125327 31139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
@ 0x7fa39c46da8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
W20240503 14:15:36.136168 13718 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.126355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:43889 (local address 127.20.240.66:34633)
0503 14:15:36.126762 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.126781 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:15:36.126895 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:15:36.135604 (+ 8709us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":2279,"threads_started":1}
W20240503 14:15:36.136265 13712 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.115359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:43889 (local address 127.20.240.70:54929)
0503 14:15:36.135633 (+ 20274us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.135734 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":20179,"spinlock_wait_cycles":11136,"thread_start_us":20163,"threads_started":1}
W20240503 14:15:36.136330 28110 consensus_peers.cc:485] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.136428 28110 consensus_peers.cc:485] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.136539 28110 consensus_peers.cc:485] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.136622 28110 consensus_peers.cc:485] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.137400 13719 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.128478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:43889 (local address 127.20.240.72:33087)
0503 14:15:36.128866 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.128884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 14:15:36.128994 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:15:36.136954 (+ 7960us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":487,"threads_started":1}
W20240503 14:15:36.137542 31139 consensus_peers.cc:485] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.137656 31139 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.137739 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.138968 30877 consensus_peers.cc:485] T 2d0b457ba3564c52824f84b78cf14c23 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.139097 30877 consensus_peers.cc:485] T 3cf751c4f28d4358b4f47c3984cfd376 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.139178 30877 consensus_peers.cc:485] T 619951b364d44766a2f3064222ceb738 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.139351 30877 consensus_peers.cc:485] T a6b4ef3ee1fd440384272cee496fc102 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.144146 28370 consensus_peers.cc:485] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.144287 30877 consensus_peers.cc:485] T bb1ec320c4114eec8915b7f2bcffc2c3 P 6bbc686598f24f518d7ca6bb2063058d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.145118 28370 consensus_peers.cc:485] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.145218 28370 consensus_peers.cc:485] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.145290 28370 consensus_peers.cc:485] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.145421 28370 consensus_peers.cc:485] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.145506 28370 consensus_peers.cc:485] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.145599 28370 consensus_peers.cc:485] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.145690 28370 consensus_peers.cc:485] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.149859 13705 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.101396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:43889 (local address 127.20.240.71:56727)
0503 14:15:36.101785 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.101805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0503 14:15:36.103205 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:15:36.145081 (+ 41876us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":2478,"threads_started":1}
W20240503 14:15:36.150177 31008 consensus_peers.cc:485] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.150324 31008 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.150410 31008 consensus_peers.cc:485] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.150483 31008 consensus_peers.cc:485] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.150568 31008 consensus_peers.cc:485] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.150643 31008 consensus_peers.cc:485] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fa39c450981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
W20240503 14:15:36.152678 31008 consensus_peers.cc:485] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.152771 31008 consensus_peers.cc:485] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:36.153751 28172 raft_consensus.cc:1203] T a88d8b316d064c4e8fdccaba77d656c2 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 14:15:36.153797 13713 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.120084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.67:43889 (local address 127.20.240.69:47231)
0503 14:15:36.153265 (+ 33181us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.153381 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33090,"thread_start_us":33009,"threads_started":1}
W20240503 14:15:36.153952 30746 consensus_peers.cc:485] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.154053 30746 consensus_peers.cc:485] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.154130 30746 consensus_peers.cc:485] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.154214 30746 consensus_peers.cc:485] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.154317 30746 consensus_peers.cc:485] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.154685 30746 consensus_peers.cc:485] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.154789 30746 consensus_peers.cc:485] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fa39c45070e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240503 14:15:36.165189 27979 consensus_peers.cc:485] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.165369 27979 consensus_peers.cc:485] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.165845 27979 consensus_peers.cc:485] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7fa38cea8b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa38ceac19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa38ceac019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fa38ce8701e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa39020c6db start_thread at ??:0
@ 0x7fa38904a71f clone at ??:0
I20240503 14:15:36.323802 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:36.329190 28041 raft_consensus.cc:1203] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:15:36.356817 28154 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:36.373502 28412 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:15:36.373515 28021 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:36.493978 31050 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 14:15:36.514065 30789 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:15:36.538116 13210 raft_consensus.cc:482] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.538296 13210 raft_consensus.cc:504] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:36.540108 13210 leader_election.cc:288] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.540733 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb01c0be7da4fe4b9c7c7160df7debc" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:36.555212 30810 raft_consensus.cc:1203] T 133f5691d69b461f8e7502ecffdb8a89 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 14:15:36.565379 30746 leader_election.cc:334] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.565521 30746 leader_election.cc:302] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.565946 13210 raft_consensus.cc:2726] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:36.578675 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:36.600313 27979 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 63724.4us
W20240503 14:15:36.651520 31138 connection.cc:535] client connection to 127.20.240.70:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.651993 31007 connection.cc:535] client connection to 127.20.240.70:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.652892 28369 connection.cc:535] client connection to 127.20.240.70:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.655484 30745 connection.cc:535] client connection to 127.20.240.70:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.656131 13705 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.652608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.70:39465 (local address 127.20.240.71:45313)
0503 14:15:36.652768 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.652786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 14:15:36.652890 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:15:36.655979 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":51}
W20240503 14:15:36.657011 13720 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.653923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.70:39465 (local address 127.20.240.68:54789)
0503 14:15:36.654067 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.654086 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:15:36.654188 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:15:36.656455 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":35}
W20240503 14:15:36.657202 28369 consensus_peers.cc:485] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657267 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:36.657299 13719 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.652710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.70:39465 (local address 127.20.240.72:32971)
0503 14:15:36.652988 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.653007 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:15:36.653130 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:15:36.657152 (+ 4022us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":177}
W20240503 14:15:36.657318 28369 consensus_peers.cc:485] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657392 30745 consensus_peers.cc:485] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657459 31138 consensus_peers.cc:485] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657509 30745 consensus_peers.cc:485] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657539 31138 consensus_peers.cc:485] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657596 31138 consensus_peers.cc:485] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657649 31138 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657711 31138 consensus_peers.cc:485] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657770 31138 consensus_peers.cc:485] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.657958 27978 connection.cc:535] client connection to 127.20.240.70:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.658957 28109 connection.cc:535] client connection to 127.20.240.70:39465 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:36.660060 31007 consensus_peers.cc:485] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.660166 31007 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.660230 31007 consensus_peers.cc:485] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662197 28109 consensus_peers.cc:485] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662317 28109 consensus_peers.cc:485] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662386 28109 consensus_peers.cc:485] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662483 28109 consensus_peers.cc:485] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662650 27978 consensus_peers.cc:485] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662760 27978 consensus_peers.cc:485] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662849 27978 consensus_peers.cc:485] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.662938 27978 consensus_peers.cc:485] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.686372 28109 consensus_peers.cc:485] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:36.690883 28109 consensus_peers.cc:485] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:36.721642 13287 raft_consensus.cc:482] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.721786 13287 raft_consensus.cc:504] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:36.722332 13287 leader_election.cc:288] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.722731 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b192797b6c40d4853e6ae4583ef3ca" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
W20240503 14:15:36.723740 31139 leader_election.cc:334] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.723846 31139 leader_election.cc:302] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.724054 13287 raft_consensus.cc:2726] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.724433 12972 raft_consensus.cc:482] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.724601 12972 raft_consensus.cc:504] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.725366 12972 leader_election.cc:288] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.726550 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583063f9427749b3aacc667ec68e55b5" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
W20240503 14:15:36.727586 28370 leader_election.cc:334] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.727718 28370 leader_election.cc:302] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.728274 12972 raft_consensus.cc:2726] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:36.728677 27978 consensus_peers.cc:485] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:36.736999 13210 raft_consensus.cc:482] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.737179 13210 raft_consensus.cc:504] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.737850 13210 leader_election.cc:288] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.738152 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
W20240503 14:15:36.741614 31007 consensus_peers.cc:485] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:36.742383 13064 raft_consensus.cc:482] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
W20240503 14:15:36.742383 30746 leader_election.cc:334] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.742516 30746 leader_election.cc:302] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:15:36.742550 13064 raft_consensus.cc:504] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:36.742798 13210 raft_consensus.cc:2726] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.743155 13064 leader_election.cc:288] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:36.744788 28109 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:36.745440 28110 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.745566 28110 leader_election.cc:302] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.745816 13064 raft_consensus.cc:2726] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.773418 13064 raft_consensus.cc:482] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.773586 13064 raft_consensus.cc:504] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:36.774219 13064 leader_election.cc:288] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:36.774855 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
W20240503 14:15:36.775619 28110 leader_election.cc:334] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.775910 28107 leader_election.cc:302] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:15:36.776158 13064 raft_consensus.cc:2726] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.783551 13287 raft_consensus.cc:482] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.783686 13287 raft_consensus.cc:504] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.784217 13287 leader_election.cc:288] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:36.789484 31138 leader_election.cc:334] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:36.790196 31139 leader_election.cc:334] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.790356 31139 leader_election.cc:302] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.790575 13792 raft_consensus.cc:2726] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.797210 13794 raft_consensus.cc:482] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.797755 13794 raft_consensus.cc:504] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.798997 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:36.799217 30810 raft_consensus.cc:2454] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:15:36.799665 31005 leader_election.cc:302] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:15:36.799926 13725 raft_consensus.cc:2781] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:36.800032 13725 raft_consensus.cc:482] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.800138 13725 raft_consensus.cc:3037] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:36.800515 31008 leader_election.cc:334] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.800781 13794 leader_election.cc:288] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.802170 13725 raft_consensus.cc:504] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.802755 13725 leader_election.cc:288] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.803287 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54a4dbdf6bd64453a9c2c78dbf550448" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:15:36.803464 30810 raft_consensus.cc:3037] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:36.806274 30810 raft_consensus.cc:2454] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:15:36.808274 31005 leader_election.cc:302] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:15:36.808645 13725 raft_consensus.cc:2781] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:36.808789 13725 raft_consensus.cc:686] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
W20240503 14:15:36.808892 31008 leader_election.cc:334] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.809044 13725 consensus_queue.cc:237] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.816275 27896 catalog_manager.cc:5484] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:36.818881 13696 raft_consensus.cc:482] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.819043 13696 raft_consensus.cc:504] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:36.819870 13696 leader_election.cc:288] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:36.820591 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf804d83954d288576c313efe3d1cb" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:36.820998 13333 raft_consensus.cc:482] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.821152 13333 raft_consensus.cc:504] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:36.821815 13333 leader_election.cc:288] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:36.822844 27979 leader_election.cc:334] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.823020 27979 leader_election.cc:302] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.823608 13696 raft_consensus.cc:2726] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:36.828696 28110 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.828784 13530 raft_consensus.cc:482] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.828953 13530 raft_consensus.cc:504] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
W20240503 14:15:36.829504 28109 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:36.829627 28109 leader_election.cc:302] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.829680 13530 leader_election.cc:288] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:36.830475 13064 raft_consensus.cc:2726] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:36.831542 28370 leader_election.cc:334] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.831614 30807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609db8c3f3b245eaa31c82915b61188f" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:36.832186 28367 leader_election.cc:302] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:15:36.832439 13530 raft_consensus.cc:2726] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.869386 13210 raft_consensus.cc:482] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.869614 13210 raft_consensus.cc:504] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:36.870330 13210 leader_election.cc:288] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.873394 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd84de9286eb4ad7b88b9f243eb2801c" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
W20240503 14:15:36.880138 30746 leader_election.cc:334] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.882263 13210 raft_consensus.cc:482] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.882416 13210 raft_consensus.cc:504] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:36.883103 13210 leader_election.cc:288] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:36.883662 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
W20240503 14:15:36.888585 30746 leader_election.cc:334] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.888739 30746 leader_election.cc:302] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f, fde134d0fa2b4900b9badd2e08fea09d
I20240503 14:15:36.889274 13210 raft_consensus.cc:2726] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.902732 30744 leader_election.cc:302] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.903199 13210 raft_consensus.cc:2726] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.904040 13530 raft_consensus.cc:482] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.904194 13530 raft_consensus.cc:504] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:36.904819 13530 leader_election.cc:288] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.905995 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf804d83954d288576c313efe3d1cb" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:36.906194 28041 raft_consensus.cc:2454] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:36.910013 28370 leader_election.cc:302] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:36.910362 12972 raft_consensus.cc:2781] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:36.910460 12972 raft_consensus.cc:482] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.910564 12972 raft_consensus.cc:3037] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:36.913379 12972 raft_consensus.cc:504] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:36.914062 12972 leader_election.cc:288] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.917965 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8baf804d83954d288576c313efe3d1cb" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:15:36.918169 28040 raft_consensus.cc:3037] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:36.920660 28040 raft_consensus.cc:2454] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 2.
W20240503 14:15:36.921224 28370 leader_election.cc:334] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.921841 28370 leader_election.cc:302] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:36.922675 13530 raft_consensus.cc:2781] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:36.922837 13530 raft_consensus.cc:686] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [term 2 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:36.923200 13530 consensus_queue.cc:237] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:36.923410 12963 raft_consensus.cc:482] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.923563 12963 raft_consensus.cc:504] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:36.924178 12963 leader_election.cc:288] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:36.925243 28370 leader_election.cc:334] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.926209 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400ce95594a743108bb4ee1e64458956" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
W20240503 14:15:36.926793 27979 leader_election.cc:334] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.927052 27979 leader_election.cc:302] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.928067 12963 raft_consensus.cc:2726] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:36.932381 12972 raft_consensus.cc:482] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.932572 12972 raft_consensus.cc:504] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:36.933746 12972 leader_election.cc:288] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:36.933853 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b192797b6c40d4853e6ae4583ef3ca" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:36.934046 31200 raft_consensus.cc:2454] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:36.935401 28368 leader_election.cc:302] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
W20240503 14:15:36.935809 28370 leader_election.cc:334] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.936769 12972 raft_consensus.cc:2781] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:36.936919 12972 raft_consensus.cc:482] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.937072 12972 raft_consensus.cc:3037] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:36.940017 12972 raft_consensus.cc:504] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
W20240503 14:15:36.942137 28370 leader_election.cc:334] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.945149 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b192797b6c40d4853e6ae4583ef3ca" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:15:36.945329 31200 raft_consensus.cc:3037] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:36.948081 31200 raft_consensus.cc:2454] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 2.
I20240503 14:15:36.948488 28368 leader_election.cc:302] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.948870 12972 leader_election.cc:288] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:36.949224 12972 raft_consensus.cc:2781] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:36.949416 12972 raft_consensus.cc:686] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [term 2 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:36.949230 27895 catalog_manager.cc:5484] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 2 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:36.949728 12972 consensus_queue.cc:237] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:36.965540 27895 catalog_manager.cc:5484] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 2 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:36.976042 12963 raft_consensus.cc:482] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.976265 12963 raft_consensus.cc:504] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.977042 12963 leader_election.cc:288] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.978868 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583063f9427749b3aacc667ec68e55b5" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:36.979071 28434 raft_consensus.cc:2454] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
W20240503 14:15:36.979908 27979 leader_election.cc:334] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.980157 27979 leader_election.cc:302] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:36.981026 12963 raft_consensus.cc:2781] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:36.981127 12963 raft_consensus.cc:482] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.981221 12963 raft_consensus.cc:3037] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:36.983271 12963 raft_consensus.cc:504] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:36.985181 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "583063f9427749b3aacc667ec68e55b5" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:36.985360 28434 raft_consensus.cc:3037] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:36.989076 28434 raft_consensus.cc:2454] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:36.989588 27979 leader_election.cc:302] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:36.989856 13757 raft_consensus.cc:2781] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:36.990018 13757 raft_consensus.cc:686] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:36.990319 13757 consensus_queue.cc:237] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
W20240503 14:15:36.990602 27979 leader_election.cc:334] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.993059 12963 leader_election.cc:288] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:36.994920 13725 raft_consensus.cc:482] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:36.995102 13725 raft_consensus.cc:504] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:36.995738 13725 leader_election.cc:288] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:36.997153 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:36.997347 30810 raft_consensus.cc:2454] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:15:36.997834 31005 leader_election.cc:302] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:15:36.998073 13725 raft_consensus.cc:2781] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:36.998180 13725 raft_consensus.cc:482] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
W20240503 14:15:36.998287 31008 leader_election.cc:334] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:36.998350 13725 raft_consensus.cc:3037] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.000721 13725 raft_consensus.cc:504] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.001259 13725 leader_election.cc:288] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:37.001695 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9202075be4bc472ebe781f577bb50ba9" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:15:37.001878 30810 raft_consensus.cc:3037] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.002628 31008 leader_election.cc:334] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.003809 13725 raft_consensus.cc:482] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.003954 13725 raft_consensus.cc:504] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.004485 30810 raft_consensus.cc:2454] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:15:37.005052 31005 leader_election.cc:302] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.005023 27893 catalog_manager.cc:5484] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:37.005329 13794 raft_consensus.cc:2781] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.005722 13725 leader_election.cc:288] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:37.005929 31008 leader_election.cc:334] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.006300 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f507cdcfb04eb4a9eb448773206783" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:37.006891 31006 leader_election.cc:302] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6409961ee8c0445687c11724213c44a9, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.007371 13725 raft_consensus.cc:2726] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.011541 13794 raft_consensus.cc:686] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:15:37.011797 13794 consensus_queue.cc:237] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.018153 27893 catalog_manager.cc:5484] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:37.025337 12392 raft_consensus.cc:482] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.025538 13530 raft_consensus.cc:482] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.025696 13530 raft_consensus.cc:504] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.025691 12392 raft_consensus.cc:504] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.026487 13530 leader_election.cc:288] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.026523 12392 leader_election.cc:288] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.027330 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609db8c3f3b245eaa31c82915b61188f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.027531 28434 raft_consensus.cc:2454] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:15:37.027648 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400ce95594a743108bb4ee1e64458956" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.027864 28040 raft_consensus.cc:2454] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:37.028928 28370 leader_election.cc:302] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:37.028975 30746 leader_election.cc:302] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:15:37.029284 12392 raft_consensus.cc:2781] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.029387 12392 raft_consensus.cc:482] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.029493 12392 raft_consensus.cc:3037] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.029534 13530 raft_consensus.cc:2781] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.029623 13530 raft_consensus.cc:482] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.029708 13530 raft_consensus.cc:3037] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.031123 28370 leader_election.cc:334] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.032203 12392 raft_consensus.cc:504] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.032742 12392 leader_election.cc:288] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.033875 13530 raft_consensus.cc:504] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
W20240503 14:15:37.035573 30746 leader_election.cc:334] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.035604 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "609db8c3f3b245eaa31c82915b61188f" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:37.036374 28434 raft_consensus.cc:3037] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.036715 13530 leader_election.cc:288] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:37.036900 30746 leader_election.cc:334] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.037699 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400ce95594a743108bb4ee1e64458956" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:15:37.037923 28040 raft_consensus.cc:3037] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.039536 28434 raft_consensus.cc:2454] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 2.
I20240503 14:15:37.040412 30746 leader_election.cc:302] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.040895 12392 raft_consensus.cc:2781] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.041580 12392 raft_consensus.cc:686] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [term 2 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:15:37.042362 12392 consensus_queue.cc:237] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
W20240503 14:15:37.045637 28370 leader_election.cc:334] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.046393 28040 raft_consensus.cc:2454] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 2.
I20240503 14:15:37.047602 28370 leader_election.cc:302] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.048388 12972 raft_consensus.cc:2781] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.048550 12972 raft_consensus.cc:686] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [term 2 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:37.048858 12972 consensus_queue.cc:237] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
W20240503 14:15:37.065258 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.065901 27893 catalog_manager.cc:5484] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 2 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:37.068532 13794 raft_consensus.cc:482] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.068702 13794 raft_consensus.cc:504] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.069411 13794 leader_election.cc:288] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:37.070781 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8868e8d0fd6b4a14970df92960ca0b80" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
W20240503 14:15:37.071537 31008 leader_election.cc:334] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.071704 31008 leader_election.cc:302] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.072028 13794 raft_consensus.cc:2726] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.074254 13210 raft_consensus.cc:482] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.074450 13210 raft_consensus.cc:504] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:37.075384 13210 leader_election.cc:288] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.077365 13757 raft_consensus.cc:482] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.077533 13757 raft_consensus.cc:504] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:37.078135 13757 leader_election.cc:288] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.079707 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611bac13cf32492b87c2f57752c59462" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.079913 28041 raft_consensus.cc:2454] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
W20240503 14:15:37.079969 27979 leader_election.cc:334] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.080456 30746 leader_election.cc:302] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:15:37.081418 13210 raft_consensus.cc:2781] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.081548 13210 raft_consensus.cc:482] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.081678 13210 raft_consensus.cc:3037] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.082212 30746 leader_election.cc:334] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.087487 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611bac13cf32492b87c2f57752c59462" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:37.088644 13210 raft_consensus.cc:504] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:37.090176 31183 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:15:37.090548 30810 raft_consensus.cc:2379] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 793a0d5dd1634c5faaf0e1e4851e8294 in current term 2: Already voted for candidate d0dbfb0e43f948c3922926aacc365fc1 in this term.
I20240503 14:15:37.092319 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "611bac13cf32492b87c2f57752c59462" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:15:37.092576 28041 raft_consensus.cc:3037] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.096299 28041 raft_consensus.cc:2454] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 2.
I20240503 14:15:37.096923 30746 leader_election.cc:302] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters:
I20240503 14:15:37.098134 13064 raft_consensus.cc:482] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.098312 13064 raft_consensus.cc:504] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.099162 13064 leader_election.cc:288] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:37.100263 31069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f507cdcfb04eb4a9eb448773206783" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:15:37.100483 31069 raft_consensus.cc:2454] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:15:37.101337 28109 leader_election.cc:302] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
W20240503 14:15:37.104398 28110 leader_election.cc:334] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.104856 13064 raft_consensus.cc:2781] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.104969 13064 raft_consensus.cc:482] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.105072 13064 raft_consensus.cc:3037] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.106246 13210 leader_election.cc:288] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.106597 13757 raft_consensus.cc:482] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.106786 13210 raft_consensus.cc:2781] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.106788 13757 raft_consensus.cc:504] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.106936 13210 raft_consensus.cc:686] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:15:37.107249 13210 consensus_queue.cc:237] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:37.107704 13757 leader_election.cc:288] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:37.108649 31069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8868e8d0fd6b4a14970df92960ca0b80" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:15:37.108924 31069 raft_consensus.cc:2454] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:15:37.110877 27978 leader_election.cc:302] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:15:37.111435 12963 raft_consensus.cc:2781] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.111550 12963 raft_consensus.cc:482] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.111649 12963 raft_consensus.cc:3037] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.112239 13812 raft_consensus.cc:482] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.112461 13812 raft_consensus.cc:504] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.113301 13812 leader_election.cc:288] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:37.114977 12963 raft_consensus.cc:504] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.115871 12963 leader_election.cc:288] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:37.117933 31069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8868e8d0fd6b4a14970df92960ca0b80" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:15:37.118183 31069 raft_consensus.cc:3037] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.118496 28110 leader_election.cc:334] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.122031 31069 raft_consensus.cc:2454] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:37.124626 27978 leader_election.cc:302] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, fde134d0fa2b4900b9badd2e08fea09d; no voters:
I20240503 14:15:37.126338 13757 raft_consensus.cc:2781] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.126688 13757 raft_consensus.cc:686] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:37.127018 13757 consensus_queue.cc:237] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
W20240503 14:15:37.127638 27979 leader_election.cc:334] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.132445 13812 raft_consensus.cc:482] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.132589 13812 raft_consensus.cc:504] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.133162 13812 leader_election.cc:288] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:37.136616 12972 raft_consensus.cc:482] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.136895 12972 raft_consensus.cc:504] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.137784 12972 leader_election.cc:288] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:37.138913 27979 leader_election.cc:334] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:37.146775 28110 leader_election.cc:334] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.149192 13064 raft_consensus.cc:504] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.150843 27893 catalog_manager.cc:5484] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:37.152083 31069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92f507cdcfb04eb4a9eb448773206783" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
W20240503 14:15:37.152247 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.152283 31069 raft_consensus.cc:3037] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.157243 28110 leader_election.cc:334] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.158944 31069 raft_consensus.cc:2454] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:15:37.158883 13841 raft_consensus.cc:482] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.158340 13064 leader_election.cc:288] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:37.159155 13841 raft_consensus.cc:504] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.159904 13064 raft_consensus.cc:482] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.160110 13064 raft_consensus.cc:504] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:37.160241 13841 leader_election.cc:288] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.160601 28109 leader_election.cc:302] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.161190 13812 raft_consensus.cc:2781] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.161470 13812 raft_consensus.cc:686] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:15:37.161845 13812 consensus_queue.cc:237] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.163231 27976 leader_election.cc:302] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294; no voters: a4cdf05256a0407c8929ac43eb55d44f, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:15:37.164053 13757 raft_consensus.cc:2726] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:37.167771 30746 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.65:33293, user_credentials={real_user=slave}} blocked reactor thread for 70878.1us
W20240503 14:15:37.171061 28109 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.173343 28110 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.173550 28110 leader_election.cc:302] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.173874 13814 raft_consensus.cc:2726] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:15:37.180011 13064 leader_election.cc:288] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:37.201834 28370 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.65:33293, user_credentials={real_user=slave}} blocked reactor thread for 150513us
W20240503 14:15:37.202418 30746 leader_election.cc:334] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:37.202737 30746 leader_election.cc:334] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.205583 13530 raft_consensus.cc:482] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.205790 13530 raft_consensus.cc:504] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.206588 13530 leader_election.cc:288] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:37.209475 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:37.209724 28173 raft_consensus.cc:2454] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
W20240503 14:15:37.210417 28370 leader_election.cc:334] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.213316 13757 raft_consensus.cc:482] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.213582 13757 raft_consensus.cc:504] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.216318 30744 leader_election.cc:302] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.217594 13841 raft_consensus.cc:2781] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240503 14:15:37.217670 27978 leader_election.cc:334] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.217725 13841 raft_consensus.cc:482] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.217833 13841 raft_consensus.cc:3037] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.218180 13757 leader_election.cc:288] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.219727 13757 raft_consensus.cc:482] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.219969 13757 raft_consensus.cc:504] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:15:37.222487 27978 leader_election.cc:334] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.223405 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:37.224118 27977 leader_election.cc:302] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.224463 12963 raft_consensus.cc:2726] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.225418 13841 raft_consensus.cc:504] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.225427 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:37.225701 28173 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:15:37.226490 13757 leader_election.cc:288] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:37.233284 13841 leader_election.cc:288] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Requested vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.233852 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea30267166942258d47155fd0173713" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:15:37.234104 28173 raft_consensus.cc:3037] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.235458 30746 leader_election.cc:334] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.238464 28173 raft_consensus.cc:2454] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 2.
I20240503 14:15:37.239007 30744 leader_election.cc:302] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.239504 13841 raft_consensus.cc:2781] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.239830 13841 raft_consensus.cc:686] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:15:37.240170 13841 consensus_queue.cc:237] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.251736 13064 raft_consensus.cc:482] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.251935 13064 raft_consensus.cc:504] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.253232 27893 catalog_manager.cc:5484] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 2 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
W20240503 14:15:37.254005 28109 leader_election.cc:334] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.259457 30810 raft_consensus.cc:1261] T 54a4dbdf6bd64453a9c2c78dbf550448 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:37.260097 13794 consensus_queue.cc:1035] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:37.262538 30789 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:37.264161 27893 catalog_manager.cc:5484] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 2 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } }
I20240503 14:15:37.264966 27893 catalog_manager.cc:5484] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 2 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:37.270015 31050 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:15:37.270298 13064 leader_election.cc:288] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:37.272104 28110 leader_election.cc:334] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.272270 28110 leader_election.cc:302] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.275485 13794 raft_consensus.cc:482] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.275635 13794 raft_consensus.cc:504] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.277022 13794 leader_election.cc:288] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.278122 13333 raft_consensus.cc:2726] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:37.278373 31007 leader_election.cc:334] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.290262 13851 raft_consensus.cc:482] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.290467 13851 raft_consensus.cc:504] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:37.293164 13851 leader_election.cc:288] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:37.293380 30745 leader_election.cc:334] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.294292 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd84de9286eb4ad7b88b9f243eb2801c" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:37.294363 30809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb01c0be7da4fe4b9c7c7160df7debc" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:37.294517 30808 raft_consensus.cc:2454] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:15:37.294556 30809 raft_consensus.cc:2454] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
I20240503 14:15:37.295466 28107 leader_election.cc:302] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.295897 13792 raft_consensus.cc:482] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.296020 13792 raft_consensus.cc:504] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.296164 28107 leader_election.cc:302] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.296559 13792 leader_election.cc:288] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:37.296772 13333 raft_consensus.cc:2781] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.296883 13333 raft_consensus.cc:482] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.297005 13333 raft_consensus.cc:3037] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.297185 30746 leader_election.cc:334] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.297302 30746 leader_election.cc:302] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.298278 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9088c0106bf249cab5b35c1171a7e7a0" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
W20240503 14:15:37.299820 31138 leader_election.cc:334] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.299840 13333 raft_consensus.cc:504] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.300686 13333 leader_election.cc:288] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:37.300915 13333 raft_consensus.cc:2781] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.301031 13333 raft_consensus.cc:482] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:37.301148 13333 raft_consensus.cc:3037] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.300694 28154 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240503 14:15:37.303186 31136 leader_election.cc:302] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:15:37.303546 13792 raft_consensus.cc:2726] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.303944 13333 raft_consensus.cc:504] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.304628 13333 leader_election.cc:288] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Requested vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:37.308341 27893 catalog_manager.cc:5484] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:37.310359 13841 raft_consensus.cc:2726] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.277788 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee914d743a2c46418f79728f3d32d1c5" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.301914 13530 raft_consensus.cc:482] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.324489 13792 raft_consensus.cc:482] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.324447 13530 raft_consensus.cc:504] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.324613 13792 raft_consensus.cc:504] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.325253 13530 leader_election.cc:288] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:37.325940 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.326144 28041 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:15:37.326908 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:37.327109 28172 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
W20240503 14:15:37.327272 31138 leader_election.cc:334] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.327656 31008 leader_election.cc:302] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.327986 13825 raft_consensus.cc:2726] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.336246 13794 raft_consensus.cc:482] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.336409 13794 raft_consensus.cc:504] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:37.337038 13794 leader_election.cc:288] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.338312 13792 leader_election.cc:288] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:37.338512 31007 leader_election.cc:334] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.339732 12963 raft_consensus.cc:482] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.339902 12963 raft_consensus.cc:504] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
W20240503 14:15:37.340345 31008 leader_election.cc:334] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.340555 31008 leader_election.cc:302] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.340649 12963 leader_election.cc:288] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.341276 13825 raft_consensus.cc:2726] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:37.342658 27978 leader_election.cc:334] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.350494 13792 raft_consensus.cc:482] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.350698 13792 raft_consensus.cc:504] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.351475 13792 leader_election.cc:288] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:37.352779 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c7ff14d43342439f935389b03ecad0" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
W20240503 14:15:37.354749 31138 leader_election.cc:334] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.354887 31138 leader_election.cc:302] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:37.355404 13792 raft_consensus.cc:2726] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:37.359225 31008 consensus_peers.cc:485] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.359920 31137 leader_election.cc:302] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.360219 13792 raft_consensus.cc:2781] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.360291 13792 raft_consensus.cc:482] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.360347 13792 raft_consensus.cc:3037] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.363149 13792 raft_consensus.cc:504] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.364146 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:15:37.364342 28041 raft_consensus.cc:3037] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.367237 28041 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 2.
I20240503 14:15:37.368283 13792 leader_election.cc:288] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:37.369148 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361b2d07d8b44d128acfb3cc941ac01a" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:15:37.369361 28172 raft_consensus.cc:3037] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.372378 28172 raft_consensus.cc:2454] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 2.
I20240503 14:15:37.372788 31137 leader_election.cc:302] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:37.373070 13792 raft_consensus.cc:2781] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.373220 13792 raft_consensus.cc:686] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [term 2 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:15:37.373808 13792 consensus_queue.cc:237] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:15:37.380322 31138 leader_election.cc:334] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.382411 13530 raft_consensus.cc:482] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.382586 13530 raft_consensus.cc:504] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.383191 13530 leader_election.cc:288] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:37.386063 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:37.386664 27898 catalog_manager.cc:5484] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 2 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:37.390446 13725 raft_consensus.cc:482] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.390611 13725 raft_consensus.cc:504] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.391261 13725 leader_election.cc:288] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:37.391863 28370 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.67:43889, user_credentials={real_user=slave}} blocked reactor thread for 181446us
I20240503 14:15:37.392191 28171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:37.392364 28171 raft_consensus.cc:2454] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 1.
I20240503 14:15:37.393062 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b82b7a5fb7c4e13b126f6d47164a463" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.393282 28041 raft_consensus.cc:2454] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:37.393573 31006 leader_election.cc:302] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters:
W20240503 14:15:37.395210 31007 leader_election.cc:334] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.395351 13725 raft_consensus.cc:2781] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.395437 13725 raft_consensus.cc:482] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.395534 13725 raft_consensus.cc:3037] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.395588 28370 leader_election.cc:334] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.396580 28370 leader_election.cc:302] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:37.396883 13839 raft_consensus.cc:2781] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.396998 13839 raft_consensus.cc:482] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.397111 13839 raft_consensus.cc:3037] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.401044 13839 raft_consensus.cc:504] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.401881 13839 leader_election.cc:288] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:37.403003 13794 raft_consensus.cc:482] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.403183 13794 raft_consensus.cc:504] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:37.403671 13725 raft_consensus.cc:504] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.403918 13794 leader_election.cc:288] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:37.404969 13725 leader_election.cc:288] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:37.405040 28172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9132d0f9d1ea4be49458a085c9e98a65" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:15:37.405216 28172 raft_consensus.cc:3037] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.405308 31007 leader_election.cc:334] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.406291 31007 leader_election.cc:334] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.407279 13841 raft_consensus.cc:482] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.407500 13841 raft_consensus.cc:504] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.408156 13841 leader_election.cc:288] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:37.408768 28172 raft_consensus.cc:2454] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fde134d0fa2b4900b9badd2e08fea09d in term 2.
I20240503 14:15:37.409229 31006 leader_election.cc:302] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.409623 13725 raft_consensus.cc:2781] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:15:37.409615 30745 leader_election.cc:334] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.409919 13725 raft_consensus.cc:686] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [term 2 LEADER]: Becoming Leader. State: Replica: fde134d0fa2b4900b9badd2e08fea09d, State: Running, Role: LEADER
I20240503 14:15:37.410200 13725 consensus_queue.cc:237] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.415755 12392 raft_consensus.cc:482] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.415997 12392 raft_consensus.cc:504] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
W20240503 14:15:37.418339 30745 leader_election.cc:334] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.419351 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a0a9b88c6a4d8ba55e2aa4662dcdbe" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.420212 30746 leader_election.cc:302] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:37.420518 13841 raft_consensus.cc:2726] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.421954 12392 leader_election.cc:288] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:37.422534 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9088c0106bf249cab5b35c1171a7e7a0" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:37.422724 31200 raft_consensus.cc:2454] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:15:37.424517 27898 catalog_manager.cc:5484] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71). New cstate: current_term: 2 leader_uuid: "fde134d0fa2b4900b9badd2e08fea09d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
W20240503 14:15:37.425369 31008 consensus_peers.cc:485] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:37.430478 28110 leader_election.cc:334] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:37.430656 28110 leader_election.cc:334] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.431857 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd84de9286eb4ad7b88b9f243eb2801c" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:15:37.432111 30810 raft_consensus.cc:3037] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.436487 13792 raft_consensus.cc:482] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.436594 30810 raft_consensus.cc:2454] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:15:37.436641 13792 raft_consensus.cc:504] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.436966 13859 raft_consensus.cc:482] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.437158 13859 raft_consensus.cc:504] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.437353 13792 leader_election.cc:288] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:37.437938 13859 leader_election.cc:288] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.438058 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.438717 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.438921 28434 raft_consensus.cc:2454] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:15:37.439332 31139 leader_election.cc:302] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:15:37.439697 13859 raft_consensus.cc:2781] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.439831 13859 raft_consensus.cc:482] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.439967 13859 raft_consensus.cc:3037] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.441843 12963 raft_consensus.cc:482] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.441990 12963 raft_consensus.cc:504] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.442620 12963 leader_election.cc:288] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:37.443660 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c7ff14d43342439f935389b03ecad0" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:37.443862 31200 raft_consensus.cc:2454] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:15:37.444269 27977 leader_election.cc:302] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:37.444515 12963 raft_consensus.cc:2781] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.444607 12963 raft_consensus.cc:482] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.444713 12963 raft_consensus.cc:3037] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.444831 13859 raft_consensus.cc:504] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.445612 13859 leader_election.cc:288] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:37.447476 31138 leader_election.cc:334] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.447644 31138 leader_election.cc:302] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.447935 13859 raft_consensus.cc:2726] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:37.448320 31138 leader_election.cc:334] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.449927 31138 leader_election.cc:334] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.450465 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6225866853d94dda88c87c6af053c739" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:37.450681 28434 raft_consensus.cc:3037] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.450625 30810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bb01c0be7da4fe4b9c7c7160df7debc" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:15:37.450863 30810 raft_consensus.cc:3037] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.453423 28434 raft_consensus.cc:2454] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 2.
I20240503 14:15:37.454082 31139 leader_election.cc:302] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.454664 30810 raft_consensus.cc:2454] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:15:37.456666 12963 raft_consensus.cc:504] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
W20240503 14:15:37.456951 27978 leader_election.cc:334] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.458442 31008 leader_election.cc:334] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.458587 31008 leader_election.cc:302] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.459115 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72c7ff14d43342439f935389b03ecad0" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:15:37.459295 31200 raft_consensus.cc:3037] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.460031 30744 leader_election.cc:302] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.460347 12392 raft_consensus.cc:2781] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.460613 12392 raft_consensus.cc:482] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
W20240503 14:15:37.460700 27978 leader_election.cc:334] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.460878 12392 raft_consensus.cc:3037] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.461275 12963 leader_election.cc:288] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:37.461534 13859 raft_consensus.cc:2781] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.461706 13859 raft_consensus.cc:686] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [term 2 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:15:37.462000 31200 raft_consensus.cc:2454] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:37.462505 27977 leader_election.cc:302] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.462693 13859 consensus_queue.cc:237] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.463019 12963 raft_consensus.cc:2781] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.463069 13725 raft_consensus.cc:2726] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.463830 12963 raft_consensus.cc:686] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:37.464318 30808 raft_consensus.cc:1261] T 9202075be4bc472ebe781f577bb50ba9 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:37.464685 12963 consensus_queue.cc:237] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.464951 13861 consensus_queue.cc:1035] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:37.468151 28107 leader_election.cc:302] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.468858 13064 raft_consensus.cc:2781] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.469201 13064 raft_consensus.cc:686] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:15:37.469601 13064 consensus_queue.cc:237] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.470372 28107 leader_election.cc:302] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, d0dbfb0e43f948c3922926aacc365fc1; no voters: a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.470831 13812 raft_consensus.cc:2781] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.470973 13812 raft_consensus.cc:686] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
W20240503 14:15:37.470983 27979 consensus_peers.cc:485] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.471253 13812 consensus_queue.cc:237] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.489691 27898 catalog_manager.cc:5484] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:37.495241 27893 catalog_manager.cc:5484] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 2 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:37.504633 12972 raft_consensus.cc:482] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.504814 12972 raft_consensus.cc:504] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.505465 12972 leader_election.cc:288] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:37.506074 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:37.508064 12972 raft_consensus.cc:482] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.508272 12972 raft_consensus.cc:504] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.509112 12972 leader_election.cc:288] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:37.514639 28370 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.65:33293, user_credentials={real_user=slave}} blocked reactor thread for 118069us
W20240503 14:15:37.515285 30744 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.72:41331, user_credentials={real_user=slave}} blocked reactor thread for 55281.8us
W20240503 14:15:37.518687 28369 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.71:34707, user_credentials={real_user=slave}} blocked reactor thread for 429183us
I20240503 14:15:37.523078 31200 raft_consensus.cc:1261] T d7b192797b6c40d4853e6ae4583ef3ca P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:37.523344 13840 raft_consensus.cc:482] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.523504 13840 raft_consensus.cc:504] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.524402 13840 leader_election.cc:288] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:37.524586 13840 consensus_queue.cc:1035] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:37.526713 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4fbfaadc474d59bfb23107f0342f7c" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.527110 13757 raft_consensus.cc:482] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.527292 13757 raft_consensus.cc:504] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.527671 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b82b7a5fb7c4e13b126f6d47164a463" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.527952 28434 raft_consensus.cc:2379] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 793a0d5dd1634c5faaf0e1e4851e8294 in current term 2: Already voted for candidate 5388902973134958a030fd63ab4faba4 in this term.
I20240503 14:15:37.527990 13757 leader_election.cc:288] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:37.530364 27978 leader_election.cc:334] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.535809 31069 raft_consensus.cc:1261] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:37.535796 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016a1c41112441619700c662beee8c98" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:37.536968 12392 raft_consensus.cc:504] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.537192 12963 consensus_queue.cc:1035] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:37.544970 27979 leader_election.cc:302] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.545254 12392 leader_election.cc:288] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:37.545538 12963 raft_consensus.cc:2726] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.546109 13840 raft_consensus.cc:482] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.546054 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b82b7a5fb7c4e13b126f6d47164a463" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:15:37.546278 28041 raft_consensus.cc:3037] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.546274 13840 raft_consensus.cc:504] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.546841 13840 leader_election.cc:288] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:37.547995 27979 leader_election.cc:302] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.548969 28041 raft_consensus.cc:2454] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 2.
I20240503 14:15:37.549190 12963 raft_consensus.cc:2726] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.549769 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5815f4c96eef4130aa0582d33a24b391" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.549865 31199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:37.549983 28434 raft_consensus.cc:2454] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:15:37.550055 31199 raft_consensus.cc:2454] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:37.550418 28368 leader_election.cc:302] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:15:37.550669 13840 raft_consensus.cc:2781] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.550770 13840 raft_consensus.cc:482] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.550863 13840 raft_consensus.cc:3037] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.550918 28369 leader_election.cc:334] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.551095 28369 leader_election.cc:334] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.551199 28369 leader_election.cc:302] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.551949 12972 raft_consensus.cc:2726] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:37.552255 28369 leader_election.cc:334] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.552482 28369 leader_election.cc:334] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.552578 28369 leader_election.cc:302] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 530893a961af4b95b6b687fa5ed22f9d, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.552796 12972 raft_consensus.cc:2726] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.552971 13814 raft_consensus.cc:482] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
W20240503 14:15:37.553073 28369 leader_election.cc:334] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.553117 13814 raft_consensus.cc:504] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.553231 28369 leader_election.cc:302] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.553432 12972 raft_consensus.cc:2726] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:37.553663 28369 leader_election.cc:334] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.553762 28369 leader_election.cc:302] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.554036 13840 raft_consensus.cc:504] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.554708 12972 raft_consensus.cc:2726] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.555114 13792 raft_consensus.cc:482] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
W20240503 14:15:37.555176 28369 leader_election.cc:334] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.555302 13530 raft_consensus.cc:482] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.555274 13792 raft_consensus.cc:504] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.555375 31199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5126a38fc740d49f638d1b3fb8a9c9" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:15:37.555452 13530 raft_consensus.cc:504] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.555552 31199 raft_consensus.cc:3037] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.555881 13792 leader_election.cc:288] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.555938 13530 leader_election.cc:288] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:37.556490 28369 leader_election.cc:334] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.556607 28369 leader_election.cc:302] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.556862 27979 leader_election.cc:302] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.557117 13757 raft_consensus.cc:2781] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.557199 13757 raft_consensus.cc:482] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.557291 13757 raft_consensus.cc:3037] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.558337 31138 leader_election.cc:334] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.560489 13840 leader_election.cc:288] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:37.561012 31199 raft_consensus.cc:2454] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 2.
I20240503 14:15:37.560988 13757 raft_consensus.cc:504] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.562387 13757 leader_election.cc:288] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:37.563028 27978 leader_election.cc:334] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.572784 13530 raft_consensus.cc:2726] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.573293 28368 leader_election.cc:302] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters:
I20240503 14:15:37.573696 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.573915 28434 raft_consensus.cc:2454] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 1.
I20240503 14:15:37.574550 12972 raft_consensus.cc:2781] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.574690 12972 raft_consensus.cc:686] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [term 2 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:37.574949 12972 consensus_queue.cc:237] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.576179 31139 leader_election.cc:302] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.576642 13792 raft_consensus.cc:2781] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.576752 13792 raft_consensus.cc:482] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.576860 13792 raft_consensus.cc:3037] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.580528 13792 raft_consensus.cc:504] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.582708 13792 leader_election.cc:288] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:37.582770 31138 leader_election.cc:334] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.582765 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b4544aefcc54a828dc3b75726367f70" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:37.582950 28434 raft_consensus.cc:3037] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.585808 28434 raft_consensus.cc:2454] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530893a961af4b95b6b687fa5ed22f9d in term 2.
I20240503 14:15:37.586408 31139 leader_election.cc:302] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.587471 13792 raft_consensus.cc:2781] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.587672 13792 raft_consensus.cc:686] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [term 2 LEADER]: Becoming Leader. State: Replica: 530893a961af4b95b6b687fa5ed22f9d, State: Running, Role: LEADER
I20240503 14:15:37.587987 13792 consensus_queue.cc:237] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
W20240503 14:15:37.600266 30745 leader_election.cc:334] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.600674 30746 consensus_peers.cc:485] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:37.604105 28369 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 47599.9us
I20240503 14:15:37.605312 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5815f4c96eef4130aa0582d33a24b391" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:37.605523 28432 raft_consensus.cc:3037] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.605556 28433 raft_consensus.cc:1261] T 583063f9427749b3aacc667ec68e55b5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:37.609095 28432 raft_consensus.cc:2454] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
W20240503 14:15:37.611078 30746 consensus_peers.cc:485] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.613207 27893 catalog_manager.cc:5484] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72). New cstate: current_term: 2 leader_uuid: "530893a961af4b95b6b687fa5ed22f9d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
I20240503 14:15:37.619324 27898 catalog_manager.cc:5484] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 2 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
W20240503 14:15:37.620016 27979 consensus_peers.cc:485] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.622480 28021 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:37.622889 28434 raft_consensus.cc:1261] T 609db8c3f3b245eaa31c82915b61188f P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:37.623559 13851 consensus_queue.cc:1035] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:37.637976 12963 raft_consensus.cc:482] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.638160 12963 raft_consensus.cc:504] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.638669 13696 raft_consensus.cc:482] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.638808 13696 raft_consensus.cc:504] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.639183 13530 raft_consensus.cc:482] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.639329 13530 raft_consensus.cc:504] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.639889 13530 leader_election.cc:288] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:37.641762 12963 leader_election.cc:288] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:37.642381 13696 leader_election.cc:288] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:37.644971 27978 leader_election.cc:334] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.645107 27978 leader_election.cc:334] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.648859 13792 raft_consensus.cc:482] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.649012 13792 raft_consensus.cc:504] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:37.649606 13792 leader_election.cc:288] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:37.656499 31139 leader_election.cc:334] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:37.656780 31138 leader_election.cc:334] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.656878 31138 leader_election.cc:302] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:37.657400 13792 raft_consensus.cc:2726] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.658135 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9088c0106bf249cab5b35c1171a7e7a0" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:15:37.658445 31200 raft_consensus.cc:3037] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.661504 31200 raft_consensus.cc:2454] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 2.
W20240503 14:15:37.674412 28110 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 82797.5us
I20240503 14:15:37.677671 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8583c4254431e8f95633c4998fc88" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.677906 28434 raft_consensus.cc:2454] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:15:37.677922 27979 leader_election.cc:302] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.682343 12963 consensus_queue.cc:1035] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:37.682776 12963 raft_consensus.cc:2781] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.682817 13814 leader_election.cc:288] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:37.682884 28173 raft_consensus.cc:1261] T dea30267166942258d47155fd0173713 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:37.682906 12963 raft_consensus.cc:686] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:37.683142 12963 consensus_queue.cc:237] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.684933 31069 raft_consensus.cc:1261] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:37.685523 13696 raft_consensus.cc:482] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.685678 13696 raft_consensus.cc:504] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.686290 13696 leader_election.cc:288] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:37.687582 28369 leader_election.cc:334] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.687848 28369 leader_election.cc:334] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:37.688067 28369 leader_election.cc:334] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.688153 13814 consensus_queue.cc:1035] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:37.689496 27978 leader_election.cc:334] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.689738 27979 leader_election.cc:302] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.690104 13696 raft_consensus.cc:2781] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.690194 13696 raft_consensus.cc:482] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.690307 13696 raft_consensus.cc:3037] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.693373 13696 raft_consensus.cc:504] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.697809 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23db2d9b3864035a08f7965bfb36574" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9" is_pre_election: true
I20240503 14:15:37.698029 28173 raft_consensus.cc:2454] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:15:37.699203 27977 leader_election.cc:302] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
W20240503 14:15:37.699467 27978 leader_election.cc:334] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.699604 13757 raft_consensus.cc:2781] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.699705 13757 raft_consensus.cc:482] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
W20240503 14:15:37.699766 28369 leader_election.cc:334] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.699803 13757 raft_consensus.cc:3037] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.701539 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8583c4254431e8f95633c4998fc88" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:37.701720 28432 raft_consensus.cc:3037] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.702569 13757 raft_consensus.cc:504] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.703099 13757 leader_election.cc:288] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
I20240503 14:15:37.703439 28412 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:37.704401 28432 raft_consensus.cc:2454] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:37.704939 27979 leader_election.cc:302] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.705174 13757 raft_consensus.cc:2781] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:15:37.713956 28109 leader_election.cc:334] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.715241 28173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23db2d9b3864035a08f7965bfb36574" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6409961ee8c0445687c11724213c44a9"
I20240503 14:15:37.715472 28173 raft_consensus.cc:3037] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.716020 13696 leader_election.cc:288] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.721571 13757 raft_consensus.cc:686] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:37.721859 13757 consensus_queue.cc:237] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.722554 28173 raft_consensus.cc:2454] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
I20240503 14:15:37.722903 27977 leader_election.cc:302] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294; no voters:
I20240503 14:15:37.723182 13884 raft_consensus.cc:2781] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.723313 13884 raft_consensus.cc:686] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:37.723549 13884 consensus_queue.cc:237] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.726392 27893 catalog_manager.cc:5484] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
W20240503 14:15:37.734818 27978 leader_election.cc:334] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.738893 27893 catalog_manager.cc:5484] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 1 to 2, leader changed from a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67) to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:37.760311 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5815f4c96eef4130aa0582d33a24b391" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.763527 28041 raft_consensus.cc:1261] T 8baf804d83954d288576c313efe3d1cb P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:37.764624 28041 raft_consensus.cc:1261] T 400ce95594a743108bb4ee1e64458956 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:37.765967 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4fbfaadc474d59bfb23107f0342f7c" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.766172 28041 raft_consensus.cc:2454] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:37.770588 28041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23db2d9b3864035a08f7965bfb36574" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.774557 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a0a9b88c6a4d8ba55e2aa4662dcdbe" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:37.774749 30808 raft_consensus.cc:2454] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 1.
I20240503 14:15:37.774741 28040 raft_consensus.cc:1261] T 611bac13cf32492b87c2f57752c59462 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 14:15:37.775204 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.775983 13841 consensus_queue.cc:1035] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:37.778508 27976 leader_election.cc:302] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.779062 13757 raft_consensus.cc:2781] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.779219 13757 raft_consensus.cc:482] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.779405 13757 raft_consensus.cc:3037] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.786418 13757 raft_consensus.cc:504] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.787654 13757 leader_election.cc:288] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Requested vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:37.791628 27978 leader_election.cc:334] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.799423 30744 leader_election.cc:302] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.800254 13844 consensus_queue.cc:1035] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:15:37.800799 13844 raft_consensus.cc:2781] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.800935 13844 raft_consensus.cc:686] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:15:37.801165 13844 consensus_queue.cc:237] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:37.801329 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98a0a9b88c6a4d8ba55e2aa4662dcdbe" candidate_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1"
I20240503 14:15:37.801468 30808 raft_consensus.cc:3037] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.801700 27898 catalog_manager.cc:5484] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:37.804633 30808 raft_consensus.cc:2454] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 793a0d5dd1634c5faaf0e1e4851e8294 in term 2.
W20240503 14:15:37.807475 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:37.811738 28040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fba8583c4254431e8f95633c4998fc88" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" is_pre_election: true
I20240503 14:15:37.816025 13842 raft_consensus.cc:482] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.816205 13842 raft_consensus.cc:504] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.816561 27898 catalog_manager.cc:5484] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:37.817111 13842 leader_election.cc:288] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:37.817144 27898 catalog_manager.cc:5484] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
W20240503 14:15:37.819339 28369 leader_election.cc:334] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.819700 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee914d743a2c46418f79728f3d32d1c5" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:15:37.819948 31070 raft_consensus.cc:2454] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:37.821743 28369 leader_election.cc:302] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.822929 13842 raft_consensus.cc:2781] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.823035 13842 raft_consensus.cc:482] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.823138 13842 raft_consensus.cc:3037] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.826977 13842 raft_consensus.cc:504] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.827785 28110 leader_election.cc:302] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:37.828013 28040 raft_consensus.cc:1261] T 361b2d07d8b44d128acfb3cc941ac01a P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:37.828109 13064 raft_consensus.cc:2726] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.828682 13859 consensus_queue.cc:1035] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:15:37.829541 28369 leader_election.cc:334] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.830345 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee914d743a2c46418f79728f3d32d1c5" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:15:37.830649 31070 raft_consensus.cc:3037] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.833618 28370 leader_election.cc:302] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.833945 13839 raft_consensus.cc:2781] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.834103 13839 raft_consensus.cc:686] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [term 2 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:37.834225 28370 leader_election.cc:302] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:37.834388 13839 consensus_queue.cc:237] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.834692 13871 raft_consensus.cc:2726] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.834818 28370 leader_election.cc:302] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.835058 13871 consensus_queue.cc:1035] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:37.835184 28370 leader_election.cc:302] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:37.835454 13530 raft_consensus.cc:2781] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.835428 13871 consensus_queue.cc:1035] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:37.835546 13530 raft_consensus.cc:482] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.835644 13530 raft_consensus.cc:3037] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.835886 13871 raft_consensus.cc:2726] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:37.836226 31070 raft_consensus.cc:2454] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 2.
I20240503 14:15:37.836910 27976 leader_election.cc:302] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.837309 12963 raft_consensus.cc:2781] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.837442 12963 raft_consensus.cc:686] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 LEADER]: Becoming Leader. State: Replica: 793a0d5dd1634c5faaf0e1e4851e8294, State: Running, Role: LEADER
I20240503 14:15:37.837684 12963 consensus_queue.cc:237] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:37.840575 13064 raft_consensus.cc:482] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.840749 13064 raft_consensus.cc:504] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:37.841398 13064 leader_election.cc:288] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:37.842883 28109 leader_election.cc:334] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.844303 13530 raft_consensus.cc:504] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.845158 13530 leader_election.cc:288] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:37.845661 28110 consensus_peers.cc:485] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.845715 28369 leader_election.cc:302] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.848140 13840 raft_consensus.cc:2781] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.848302 13840 raft_consensus.cc:686] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [term 2 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:37.848582 13840 consensus_queue.cc:237] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:37.850471 28432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016a1c41112441619700c662beee8c98" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:37.850687 28432 raft_consensus.cc:2454] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 1.
W20240503 14:15:37.851950 28369 leader_election.cc:334] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.858330 13842 leader_election.cc:288] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:37.868062 27893 catalog_manager.cc:5484] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 2 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
W20240503 14:15:37.869314 28370 consensus_peers.cc:485] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:37.869447 28370 consensus_peers.cc:485] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:37.869562 28370 consensus_peers.cc:485] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.870173 28042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4fbfaadc474d59bfb23107f0342f7c" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "793a0d5dd1634c5faaf0e1e4851e8294"
I20240503 14:15:37.870430 28042 raft_consensus.cc:3037] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:37.873759 30746 consensus_peers.cc:485] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.873823 28042 raft_consensus.cc:2454] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 2.
I20240503 14:15:37.886929 28110 leader_election.cc:302] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.887495 13876 raft_consensus.cc:2781] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:37.887607 13876 raft_consensus.cc:482] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:37.887703 13876 raft_consensus.cc:3037] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.891558 27898 catalog_manager.cc:5484] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 2 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:37.893105 13876 raft_consensus.cc:504] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:15:37.898628 28109 leader_election.cc:334] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:37.900619 13876 leader_election.cc:288] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Requested vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:37.903717 28370 leader_election.cc:302] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.905864 27898 catalog_manager.cc:5484] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 2 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } }
I20240503 14:15:37.906448 28434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016a1c41112441619700c662beee8c98" candidate_uuid: "6409961ee8c0445687c11724213c44a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4"
I20240503 14:15:37.906647 28434 raft_consensus.cc:3037] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:37.908352 13839 raft_consensus.cc:2781] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.908488 13839 raft_consensus.cc:686] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [term 2 LEADER]: Becoming Leader. State: Replica: 5388902973134958a030fd63ab4faba4, State: Running, Role: LEADER
I20240503 14:15:37.908741 13839 consensus_queue.cc:237] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:37.909437 28434 raft_consensus.cc:2454] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6409961ee8c0445687c11724213c44a9 in term 2.
I20240503 14:15:37.914355 27893 catalog_manager.cc:5484] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 5388902973134958a030fd63ab4faba4 (127.20.240.68). New cstate: current_term: 2 leader_uuid: "5388902973134958a030fd63ab4faba4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } }
I20240503 14:15:37.914271 28110 leader_election.cc:302] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5388902973134958a030fd63ab4faba4, 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:37.914940 13814 raft_consensus.cc:2781] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:37.915130 13814 raft_consensus.cc:686] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [term 2 LEADER]: Becoming Leader. State: Replica: 6409961ee8c0445687c11724213c44a9, State: Running, Role: LEADER
I20240503 14:15:37.915412 13814 consensus_queue.cc:237] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:15:37.923741 31138 consensus_peers.cc:485] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:37.926005 27898 catalog_manager.cc:5484] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } }
I20240503 14:15:37.927482 28173 raft_consensus.cc:1261] T 9132d0f9d1ea4be49458a085c9e98a65 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer fde134d0fa2b4900b9badd2e08fea09d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:37.928031 13861 consensus_queue.cc:1035] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:37.927379 27893 catalog_manager.cc:5484] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 reported cstate change: term changed from 1 to 2, leader changed from 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70) to 6409961ee8c0445687c11724213c44a9 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "6409961ee8c0445687c11724213c44a9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } }
I20240503 14:15:37.933360 28434 raft_consensus.cc:1261] T 6225866853d94dda88c87c6af053c739 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:37.934058 13792 consensus_queue.cc:1035] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:37.951090 28172 raft_consensus.cc:1261] T 361b2d07d8b44d128acfb3cc941ac01a P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:15:37.951716 13792 consensus_queue.cc:1035] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:15:37.956108 31138 consensus_peers.cc:485] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:37.973141 31138 consensus_peers.cc:485] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:37.975615 27979 consensus_peers.cc:485] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:37.986501 30746 consensus_peers.cc:485] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:37.988116 31200 raft_consensus.cc:1261] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:37.988672 13839 consensus_queue.cc:1035] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:38.005241 31007 consensus_peers.cc:485] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.007366 31200 raft_consensus.cc:1261] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240503 14:15:38.008312 28369 consensus_peers.cc:485] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.008322 12963 consensus_queue.cc:1035] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:38.018643 27978 consensus_peers.cc:485] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:38.077329 27978 consensus_peers.cc:485] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.078465 28434 raft_consensus.cc:1261] T 1b4544aefcc54a828dc3b75726367f70 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 530893a961af4b95b6b687fa5ed22f9d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:38.079025 13859 consensus_queue.cc:1035] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:38.082935 28433 raft_consensus.cc:1261] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:38.083654 13884 consensus_queue.cc:1035] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:38.130365 27978 consensus_peers.cc:485] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.144857 30808 raft_consensus.cc:1261] T cd84de9286eb4ad7b88b9f243eb2801c P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240503 14:15:38.147236 28110 consensus_peers.cc:485] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.147818 13812 consensus_queue.cc:1035] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:15:38.151906 31008 consensus_peers.cc:485] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.152513 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:38.155531 28110 consensus_peers.cc:485] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:38.196717 27978 consensus_peers.cc:485] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.214777 28173 raft_consensus.cc:1261] T a23db2d9b3864035a08f7965bfb36574 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:38.215598 31200 raft_consensus.cc:1261] T 9088c0106bf249cab5b35c1171a7e7a0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:38.215574 13885 consensus_queue.cc:1035] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:15:38.216256 13892 consensus_queue.cc:1035] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:38.222226 30745 consensus_peers.cc:485] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.265740 30808 raft_consensus.cc:1261] T 1bb01c0be7da4fe4b9c7c7160df7debc P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:15:38.266448 13812 consensus_queue.cc:1035] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:15:38.283221 28370 consensus_peers.cc:485] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.285310 31139 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.287490 28043 raft_consensus.cc:1261] T 7b82b7a5fb7c4e13b126f6d47164a463 P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:38.288656 13871 consensus_queue.cc:1035] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:38.297389 28369 consensus_peers.cc:485] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:38.305763 31008 consensus_peers.cc:485] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.306672 30746 consensus_peers.cc:485] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.312716 28041 raft_consensus.cc:1261] T 1f4fbfaadc474d59bfb23107f0342f7c P 793a0d5dd1634c5faaf0e1e4851e8294 [term 2 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:15:38.313836 13530 consensus_queue.cc:1035] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:38.326769 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:38.330309 28434 raft_consensus.cc:1261] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:15:38.333686 28412 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 14:15:38.344383 13910 raft_consensus.cc:482] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:38.344534 13910 raft_consensus.cc:504] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:38.345137 13910 leader_election.cc:288] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:38.347209 13884 consensus_queue.cc:1035] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:38.348470 31008 consensus_peers.cc:485] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.349385 31069 raft_consensus.cc:1261] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Refusing update from remote peer 5388902973134958a030fd63ab4faba4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240503 14:15:38.349996 31139 leader_election.cc:334] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:38.353008 13530 consensus_queue.cc:1035] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:38.358595 30808 raft_consensus.cc:1261] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Refusing update from remote peer 793a0d5dd1634c5faaf0e1e4851e8294: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 14:15:38.358763 27978 consensus_peers.cc:485] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.359808 13757 consensus_queue.cc:1035] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:15:38.379344 31138 leader_election.cc:334] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:38.379555 31138 leader_election.cc:302] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:38.380532 13910 raft_consensus.cc:2726] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:38.384286 27979 consensus_peers.cc:485] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.384613 28370 consensus_peers.cc:485] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.385943 28110 consensus_peers.cc:485] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.389704 28109 consensus_peers.cc:485] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:38.399363 31183 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:15:38.400131 28154 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:15:38.416921 28370 consensus_peers.cc:485] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.432049 28369 consensus_peers.cc:485] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:38.433725 28369 consensus_peers.cc:485] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:38.434489 31138 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 51735us
I20240503 14:15:38.445891 28021 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:38.454929 30789 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:15:38.455085 31139 consensus_peers.cc:485] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.467964 30746 consensus_peers.cc:485] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.483654 30745 consensus_peers.cc:485] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.521347 31008 consensus_peers.cc:485] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.524132 28433 raft_consensus.cc:1261] T 016a1c41112441619700c662beee8c98 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Refusing update from remote peer 6409961ee8c0445687c11724213c44a9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240503 14:15:38.529160 30745 consensus_peers.cc:485] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.529446 28110 consensus_peers.cc:485] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.529891 13812 consensus_queue.cc:1035] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:15:38.538067 28110 consensus_peers.cc:485] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.578833 28370 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.67:43889, user_credentials={real_user=slave}} blocked reactor thread for 162004us
I20240503 14:15:38.580052 31050 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:15:38.581764 30746 consensus_peers.cc:485] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.585081 31008 consensus_peers.cc:485] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.588363 30746 consensus_peers.cc:485] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.589195 28370 consensus_peers.cc:485] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.592950 28370 consensus_peers.cc:485] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.595548 13814 raft_consensus.cc:482] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:38.595695 13814 raft_consensus.cc:504] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:38.596284 13814 leader_election.cc:288] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:38.598999 28370 consensus_peers.cc:485] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.623427 28109 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:38.635339 27979 consensus_peers.cc:485] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.638315 31008 consensus_peers.cc:485] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.642616 30746 consensus_peers.cc:485] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.646900 28370 consensus_peers.cc:485] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.650686 28110 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:38.650830 28110 leader_election.cc:302] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:38.651063 13814 raft_consensus.cc:2726] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:38.662246 31008 consensus_peers.cc:485] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.692802 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:38.707012 13876 raft_consensus.cc:482] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:38.707185 13876 raft_consensus.cc:504] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:38.707846 13876 leader_election.cc:288] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:38.709374 28109 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:38.709517 28110 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:38.709619 28110 leader_election.cc:302] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:38.709964 13876 raft_consensus.cc:2726] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:38.773104 31138 consensus_peers.cc:485] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.824411 30746 consensus_peers.cc:485] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.835261 28109 consensus_peers.cc:485] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.857049 13851 raft_consensus.cc:482] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:38.857283 13851 raft_consensus.cc:504] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:38.858183 13851 leader_election.cc:288] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:38.859760 30745 leader_election.cc:334] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:38.860983 31138 consensus_peers.cc:485] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.863224 13825 raft_consensus.cc:482] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:38.863361 13825 raft_consensus.cc:504] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:38.863957 13825 leader_election.cc:288] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:38.864356 31008 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.868219 31007 leader_election.cc:334] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:38.870150 30746 leader_election.cc:334] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:38.870307 30746 leader_election.cc:302] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:38.870743 13843 raft_consensus.cc:2726] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:38.870962 31008 leader_election.cc:334] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:38.871062 31008 leader_election.cc:302] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:38.871291 13861 raft_consensus.cc:2726] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:38.873273 28110 consensus_peers.cc:485] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.892120 31138 consensus_peers.cc:485] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.900177 28109 consensus_peers.cc:485] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.909780 27978 consensus_peers.cc:485] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.912189 27978 consensus_peers.cc:485] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.934255 27978 consensus_peers.cc:485] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.938575 31138 consensus_peers.cc:485] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.944707 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:38.950183 31138 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:38.991967 31138 consensus_peers.cc:485] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:38.998914 31183 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:15:39.023188 28412 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 14:15:39.036101 28370 consensus_peers.cc:485] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.037034 13814 raft_consensus.cc:482] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:39.037233 13814 raft_consensus.cc:504] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:39.038071 13814 leader_election.cc:288] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:39.040040 28110 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:39.050470 28109 consensus_peers.cc:485] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.051339 28109 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:39.051468 28109 leader_election.cc:302] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:39.053524 13814 raft_consensus.cc:2726] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:39.057360 30745 consensus_peers.cc:485] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.065409 28154 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:39.073397 31050 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:15:39.073693 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:39.074380 28021 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:39.076928 30789 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:15:39.087368 28369 consensus_peers.cc:485] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.096686 31007 consensus_peers.cc:485] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.109777 31007 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.161927 27978 consensus_peers.cc:485] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.201252 31007 consensus_peers.cc:485] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.216208 13825 raft_consensus.cc:482] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:39.216365 13825 raft_consensus.cc:504] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:39.216958 13825 leader_election.cc:288] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:39.218295 31008 leader_election.cc:334] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:39.218593 31007 leader_election.cc:334] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:39.218731 31007 leader_election.cc:302] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:39.219221 13825 raft_consensus.cc:2726] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:39.301932 13859 raft_consensus.cc:482] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:39.302109 13859 raft_consensus.cc:504] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:39.302696 13859 leader_election.cc:288] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:39.304023 31138 leader_election.cc:334] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:39.304700 31139 leader_election.cc:334] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:39.304816 31139 leader_election.cc:302] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:39.305169 13859 raft_consensus.cc:2726] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:39.309445 28369 consensus_peers.cc:485] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.321233 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:39.322535 27978 consensus_peers.cc:485] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.333009 28109 consensus_peers.cc:485] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.346721 28109 consensus_peers.cc:485] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.405681 13530 raft_consensus.cc:482] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:39.405925 13530 raft_consensus.cc:504] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:39.406678 13530 leader_election.cc:288] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:39.408668 28370 leader_election.cc:334] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:39.408792 28369 leader_election.cc:334] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:39.408885 28369 leader_election.cc:302] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
W20240503 14:15:39.409396 31007 consensus_peers.cc:485] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.409876 13530 raft_consensus.cc:2726] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:39.475677 31183 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:15:39.477953 28412 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 14:15:39.478966 13814 raft_consensus.cc:482] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:39.479137 13814 raft_consensus.cc:504] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:39.480037 13814 leader_election.cc:288] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:39.483413 28109 leader_election.cc:334] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:39.484135 28110 leader_election.cc:334] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:39.484254 28110 leader_election.cc:302] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:39.484620 13814 raft_consensus.cc:2726] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:39.497360 28154 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:15:39.506778 28109 consensus_peers.cc:485] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.528546 28021 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:39.538038 31050 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:39.555259 30789 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:15:39.577078 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:39.679322 13530 raft_consensus.cc:482] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:39.679487 13530 raft_consensus.cc:504] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:39.680341 13530 leader_election.cc:288] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:39.682969 28370 leader_election.cc:334] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:39.683178 28369 leader_election.cc:334] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:39.683284 28369 leader_election.cc:302] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:39.683697 13839 raft_consensus.cc:2726] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:39.704368 28370 consensus_peers.cc:485] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.816154 31008 consensus_peers.cc:485] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.835510 30746 consensus_peers.cc:485] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:39.855394 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:39.906183 30746 consensus_peers.cc:485] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.936407 28412 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240503 14:15:39.938501 31183 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240503 14:15:39.959277 31008 consensus_peers.cc:485] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:39.974190 28021 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:39.988193 28154 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:39.999563 31050 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:15:40.005255 27979 consensus_peers.cc:485] T 583063f9427749b3aacc667ec68e55b5 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:40.026192 30789 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:15:40.037393 28110 consensus_peers.cc:485] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.038178 28370 consensus_peers.cc:485] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.056792 27979 consensus_peers.cc:485] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.137948 28370 consensus_peers.cc:485] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.140645 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:40.148316 30746 consensus_peers.cc:485] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:40.246172 13814 raft_consensus.cc:482] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:40.246412 13814 raft_consensus.cc:504] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:40.247089 13814 leader_election.cc:288] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:40.251423 28110 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:40.252908 28109 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:40.279877 28109 leader_election.cc:302] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:40.280645 13814 raft_consensus.cc:2726] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:40.290094 31138 consensus_peers.cc:485] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.297139 31138 consensus_peers.cc:485] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.302762 31138 consensus_peers.cc:485] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.322280 31007 consensus_peers.cc:485] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.363448 27979 consensus_peers.cc:485] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:40.415179 31183 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:15:40.427249 28412 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 14:15:40.433372 30746 consensus_peers.cc:485] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.435663 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:40.451393 28154 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:15:40.460563 30746 consensus_peers.cc:485] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:40.490307 28021 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:40.502775 30789 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240503 14:15:40.504439 13814 raft_consensus.cc:482] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:40.504606 13814 raft_consensus.cc:504] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:40.505362 13814 leader_election.cc:288] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:40.509007 28109 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:40.517182 28110 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:40.517299 28110 leader_election.cc:302] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:40.517546 13814 raft_consensus.cc:2726] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:40.552554 31050 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:15:40.584590 28110 consensus_peers.cc:485] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.586637 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:40.598811 28110 consensus_peers.cc:485] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.613788 31139 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.623755 27978 consensus_peers.cc:485] T 72c7ff14d43342439f935389b03ecad0 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.631789 28369 consensus_peers.cc:485] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.668390 28369 consensus_peers.cc:485] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.678387 30745 consensus_peers.cc:485] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.678822 28370 consensus_peers.cc:485] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.692617 27978 consensus_peers.cc:485] T 5815f4c96eef4130aa0582d33a24b391 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.693297 27978 consensus_peers.cc:485] T fba8583c4254431e8f95633c4998fc88 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.698745 31008 consensus_peers.cc:485] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.720044 31008 consensus_peers.cc:485] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.747170 28369 consensus_peers.cc:485] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.772662 28109 consensus_peers.cc:485] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.776973 31139 consensus_peers.cc:485] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.778537 30745 consensus_peers.cc:485] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:40.798571 13814 raft_consensus.cc:482] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:40.798739 13814 raft_consensus.cc:504] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:40.799402 13814 leader_election.cc:288] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:40.805979 28109 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:40.809734 28110 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:40.809827 28110 leader_election.cc:302] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:40.810071 13876 raft_consensus.cc:2726] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:40.855753 28369 consensus_peers.cc:485] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.859716 28370 consensus_peers.cc:485] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:40.860730 14209 raft_consensus.cc:482] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:40.860911 14209 raft_consensus.cc:504] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:40.861622 14209 leader_election.cc:288] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:40.867223 31008 consensus_peers.cc:485] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.870347 31139 leader_election.cc:334] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:40.870673 31138 leader_election.cc:334] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:40.870770 31138 leader_election.cc:302] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:40.871111 14209 raft_consensus.cc:2726] T 0a6b0218ec01450f94a394efb7d82149 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:40.900122 28370 consensus_peers.cc:485] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.922668 27978 consensus_peers.cc:485] T a23db2d9b3864035a08f7965bfb36574 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:15:40.924086 28370 consensus_peers.cc:485] T edd300244be74e3aa94d7ae35deeb470 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:40.959892 28154 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:15:40.967519 31008 consensus_peers.cc:485] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:40.972003 28021 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
I20240503 14:15:40.973167 14209 raft_consensus.cc:482] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:40.973354 14209 raft_consensus.cc:504] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:40.974020 14209 leader_election.cc:288] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:40.977152 31139 leader_election.cc:334] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:40.977371 31138 leader_election.cc:334] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:40.977486 31138 leader_election.cc:302] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:40.977716 14209 raft_consensus.cc:2726] T 3cf751c4f28d4358b4f47c3984cfd376 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:40.979910 28370 consensus_peers.cc:485] T 48266bf4ad9b4976b639553490ffebbe P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:40.981146 30745 consensus_peers.cc:485] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:40.986019 31050 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:15:40.989809 30789 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:15:40.990756 27978 consensus_peers.cc:485] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:15:40.997709 31183 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:15:41.017386 28412 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 14:15:41.031859 27978 consensus_peers.cc:485] T dc9c29880eea4cd9a11eb24eeb3bd228 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.044039 13814 raft_consensus.cc:482] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:41.044194 13814 raft_consensus.cc:504] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
W20240503 14:15:41.044391 27979 consensus_peers.cc:485] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.044890 13814 leader_election.cc:288] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:41.048243 31008 consensus_peers.cc:485] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.048767 28109 leader_election.cc:334] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:41.050040 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:41.052076 27979 consensus_peers.cc:485] T 06e418c4c2cf46608ba454d48fae9285 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.054461 28110 leader_election.cc:334] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:41.054586 28110 leader_election.cc:302] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:41.054941 13814 raft_consensus.cc:2726] T a6b4ef3ee1fd440384272cee496fc102 P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:41.056389 30746 consensus_peers.cc:485] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.059163 31138 consensus_peers.cc:485] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.079738 30746 consensus_peers.cc:485] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.080471 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:41.083467 31138 consensus_peers.cc:485] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.089928 28110 consensus_peers.cc:485] T 30e4fb0a79814959aa25cf6eb34dfbb9 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.092826 31007 consensus_peers.cc:485] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.093312 28370 consensus_peers.cc:485] T 69d4d86cb68646bc9141c1e6df964561 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.105324 31007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111) [suppressed 163 similar messages]
W20240503 14:15:41.123607 28369 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111) [suppressed 179 similar messages]
W20240503 14:15:41.125185 31138 consensus_peers.cc:485] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.142697 28110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111) [suppressed 160 similar messages]
W20240503 14:15:41.143467 28110 consensus_peers.cc:485] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.174996 13530 raft_consensus.cc:482] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:41.175158 13530 raft_consensus.cc:504] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:41.175755 13530 leader_election.cc:288] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:41.178085 28370 leader_election.cc:334] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:41.178877 28369 leader_election.cc:334] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:41.178978 28369 leader_election.cc:302] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:41.179505 13839 raft_consensus.cc:2726] T 16c2801d0bf442519ab3841a98d37696 P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:41.182587 30746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111) [suppressed 161 similar messages]
W20240503 14:15:41.188182 31008 consensus_peers.cc:485] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.197767 28370 consensus_peers.cc:485] T 213bc70b5cf14440ab88c754df0b8f7f P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.213158 31139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 14:15:41.223053 27978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111) [suppressed 150 similar messages]
W20240503 14:15:41.251425 30746 consensus_peers.cc:485] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.253988 30746 consensus_peers.cc:485] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.266005 31008 consensus_peers.cc:485] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.285334 28110 consensus_peers.cc:485] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.298970 30746 consensus_peers.cc:485] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.345952 28110 consensus_peers.cc:485] T dcb14382755441afb4cc5c103425331c P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.348912 31138 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.370038 27978 consensus_peers.cc:485] T cb88c79b4b1a40c2b45c2dd79ab894d4 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.379909 31138 consensus_peers.cc:485] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.412932 27978 consensus_peers.cc:485] T 344eb942dc3d4188bc9baf3d1ecc4f7a P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.469915 28109 consensus_peers.cc:485] T 38cf3aab9e1b4f28842ac16d02be2562 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.470367 31007 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.475894 31008 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.488318 28369 consensus_peers.cc:485] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.489073 28109 consensus_peers.cc:485] T 45da3dab7a204d378273a8985d91ace8 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.491492 27978 consensus_peers.cc:485] T 6b263006614c46b4ac0648efb7a37a8f P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.519812 31138 consensus_peers.cc:485] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.539417 28412 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240503 14:15:41.546914 28109 consensus_peers.cc:485] T 02559d1ab4824298ba76a44a7f05303b P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.549278 13825 raft_consensus.cc:482] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:41.549450 13825 raft_consensus.cc:504] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:41.550194 13825 leader_election.cc:288] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:41.553475 31008 leader_election.cc:334] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:41.555080 31183 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:15:41.571585 28154 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:15:41.601176 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:41.633131 27978 consensus_peers.cc:485] T 5f25a3bcdef849499e4a02f01f4dbc0e P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.635675 31007 leader_election.cc:334] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:41.635843 31007 leader_election.cc:302] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:41.636168 13825 raft_consensus.cc:2726] T bb1ec320c4114eec8915b7f2bcffc2c3 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:41.653558 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:41.657631 28370 consensus_peers.cc:485] T 823dde634aac449ea8cd21d38ecfb22b P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.689250 28021 tablet_service.cc:1429] Tablet server has 23 leaders and 0 scanners
W20240503 14:15:41.696331 28109 consensus_peers.cc:485] T f7bdba20d1844eefb6d3b29c5deb2873 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.697293 30789 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240503 14:15:41.728384 30745 consensus_peers.cc:485] T f4b9d916f3fb4a348a8702662c143435 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.739460 31050 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240503 14:15:41.755746 31007 consensus_peers.cc:485] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.768249 28369 consensus_peers.cc:485] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:41.780472 28109 consensus_peers.cc:485] T a82470f287274b1c878089fa9d56ef40 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:41.944509 14354 raft_consensus.cc:482] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:41.944689 14354 raft_consensus.cc:504] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:41.945353 14354 leader_election.cc:288] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:41.946770 31007 leader_election.cc:334] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:41.946902 31008 leader_election.cc:334] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:41.947046 31008 leader_election.cc:302] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:41.947304 14354 raft_consensus.cc:2726] T 2d0b457ba3564c52824f84b78cf14c23 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:42.005018 28109 consensus_peers.cc:485] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.158128 31007 consensus_peers.cc:485] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.209262 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:42.209285 28370 consensus_peers.cc:485] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.213804 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:42.228771 14358 raft_consensus.cc:482] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:42.228919 14358 raft_consensus.cc:504] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:42.229439 14358 leader_election.cc:288] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
I20240503 14:15:42.230295 13530 raft_consensus.cc:482] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
W20240503 14:15:42.231137 30746 leader_election.cc:334] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:42.231354 30745 leader_election.cc:334] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:42.231479 30745 leader_election.cc:302] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:42.231736 14358 raft_consensus.cc:2726] T 619951b364d44766a2f3064222ceb738 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:42.231850 13530 raft_consensus.cc:504] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:42.233099 13530 leader_election.cc:288] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:42.233652 28370 leader_election.cc:334] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:42.233799 28369 leader_election.cc:334] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:42.233927 28369 leader_election.cc:302] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5388902973134958a030fd63ab4faba4; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:42.234148 13839 raft_consensus.cc:2726] T 8377b1aa704c43b78f310e7440f6488e P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:42.272028 28110 consensus_peers.cc:485] T 92f507cdcfb04eb4a9eb448773206783 P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.349941 30746 consensus_peers.cc:485] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:42.350394 13876 raft_consensus.cc:482] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:42.350567 13876 raft_consensus.cc:504] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:42.351250 13876 leader_election.cc:288] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:42.352768 28109 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:42.352952 28110 leader_election.cc:334] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:42.353081 28110 leader_election.cc:302] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:42.353401 13876 raft_consensus.cc:2726] T f133ebe695a74f86be1a37ae661719ec P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:42.367130 31008 consensus_peers.cc:485] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.381163 27979 consensus_peers.cc:485] T 8868e8d0fd6b4a14970df92960ca0b80 P 793a0d5dd1634c5faaf0e1e4851e8294 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 48
Maximum Replica Count | 51
Average Replica Count | 49.875000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.511111
stderr: W20240503 14:13:56.832183 561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:56.832546 561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:56.843269 561 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:58.267397 561 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240503 14:13:59.169943 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2498fceee67f406a8a21f9f8c19b426e: could not find any suitable replica to move from server 530893a961af4b95b6b687fa5ed22f9d to server 5388902973134958a030fd63ab4faba4
W20240503 14:13:59.170341 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 00386c23ce26494485b6cc25f3ff5689: could not find any suitable replica to move from server 5388902973134958a030fd63ab4faba4 to server 530893a961af4b95b6b687fa5ed22f9d
W20240503 14:13:59.170640 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ceeff6e8ae044b7c8fc0d7663df1b1a9: could not find any suitable replica to move from server fde134d0fa2b4900b9badd2e08fea09d to server 6409961ee8c0445687c11724213c44a9
W20240503 14:13:59.170789 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 914cf0996edd4e549f6daa8a3210a29d: could not find any suitable replica to move from server a4cdf05256a0407c8929ac43eb55d44f to server fde134d0fa2b4900b9badd2e08fea09d
W20240503 14:13:59.171435 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0672b9d4f2f94999b19c7e1b2a2c7b12: could not find any suitable replica to move from server 6bbc686598f24f518d7ca6bb2063058d to server fde134d0fa2b4900b9badd2e08fea09d
W20240503 14:13:59.171814 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 06e77778ef6142a5bfa4db225f6cb559: could not find any suitable replica to move from server 6409961ee8c0445687c11724213c44a9 to server 793a0d5dd1634c5faaf0e1e4851e8294
W20240503 14:13:59.172215 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 548f97bb3bcd4f4bb98698fc00bc1ca5: could not find any suitable replica to move from server 530893a961af4b95b6b687fa5ed22f9d to server fde134d0fa2b4900b9badd2e08fea09d
W20240503 14:13:59.172358 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3744f81450ee4674a77d5671540dc86f: could not find any suitable replica to move from server fde134d0fa2b4900b9badd2e08fea09d to server 6409961ee8c0445687c11724213c44a9
W20240503 14:13:59.172544 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4673aaf684274f249d74ffa7649ceb73: could not find any suitable replica to move from server 6409961ee8c0445687c11724213c44a9 to server fde134d0fa2b4900b9badd2e08fea09d
W20240503 14:13:59.172672 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4a94574e1a5b44909f1e81be5a6767f9: could not find any suitable replica to move from server a4cdf05256a0407c8929ac43eb55d44f to server 530893a961af4b95b6b687fa5ed22f9d
W20240503 14:13:59.172766 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9c30e6ab7d16493493d8c70e665de3fd: could not find any suitable replica to move from server 6409961ee8c0445687c11724213c44a9 to server 530893a961af4b95b6b687fa5ed22f9d
W20240503 14:13:59.172884 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 698fdaaae5954d1b8048ce09e5653803: could not find any suitable replica to move from server 5388902973134958a030fd63ab4faba4 to server 530893a961af4b95b6b687fa5ed22f9d
W20240503 14:13:59.173002 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ca9ae4dcdb734b4caba55748be1857a2: could not find any suitable replica to move from server 6409961ee8c0445687c11724213c44a9 to server 6bbc686598f24f518d7ca6bb2063058d
W20240503 14:13:59.173116 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 06682f622ac341c293100bd740ae19cc: could not find any suitable replica to move from server 6bbc686598f24f518d7ca6bb2063058d to server 793a0d5dd1634c5faaf0e1e4851e8294
W20240503 14:13:59.173219 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 91d24d2ac8f7438fa2967ca61fb59b9d: could not find any suitable replica to move from server fde134d0fa2b4900b9badd2e08fea09d to server 793a0d5dd1634c5faaf0e1e4851e8294
W20240503 14:13:59.173345 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6639b1cb53f44be0b930873e71075e93: could not find any suitable replica to move from server 793a0d5dd1634c5faaf0e1e4851e8294 to server 6bbc686598f24f518d7ca6bb2063058d
W20240503 14:13:59.173486 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 532b45778ec745f6a99ffe981dd1eec5: could not find any suitable replica to move from server 793a0d5dd1634c5faaf0e1e4851e8294 to server fde134d0fa2b4900b9badd2e08fea09d
W20240503 14:13:59.173651 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f1a0906eb4d644e88b1eb9ef75dc7b19: could not find any suitable replica to move from server a4cdf05256a0407c8929ac43eb55d44f to server d0dbfb0e43f948c3922926aacc365fc1
W20240503 14:13:59.173962 561 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1e1dcbe104d549689601d3ca207c3e59: could not find any suitable replica to move from server 6bbc686598f24f518d7ca6bb2063058d to server a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:13:59.245507 561 rebalancer_tool.cc:1074] tablet 45da3dab7a204d378273a8985d91ace8: 'd0dbfb0e43f948c3922926aacc365fc1' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:13:59.317963 561 rebalancer_tool.cc:1074] tablet 9a49047eedc54168af70a7b59fd7d770: 'fde134d0fa2b4900b9badd2e08fea09d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:13:59.391337 561 rebalancer_tool.cc:1074] tablet a6b4ef3ee1fd440384272cee496fc102: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:13:59.452338 561 rebalancer_tool.cc:1074] tablet a3888675106d4d389718380627c95edf: '5388902973134958a030fd63ab4faba4' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:13:59.519266 561 rebalancer_tool.cc:1074] tablet 344eb942dc3d4188bc9baf3d1ecc4f7a: 'd0dbfb0e43f948c3922926aacc365fc1' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:13:59.583626 561 rebalancer_tool.cc:1074] tablet 9517b56a435b48e18c79d2a375e5a951: 'fde134d0fa2b4900b9badd2e08fea09d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:13:59.701328 561 rebalancer_tool.cc:1074] tablet 016a1c41112441619700c662beee8c98: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:13:59.786765 561 rebalancer_tool.cc:1074] tablet cd84de9286eb4ad7b88b9f243eb2801c: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:13:59.862421 561 rebalancer_tool.cc:1074] tablet f4b9d916f3fb4a348a8702662c143435: 'a4cdf05256a0407c8929ac43eb55d44f' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:13:59.997720 561 rebalancer_tool.cc:1074] tablet 618ba33946424065a465217bae7cdbb0: 'fde134d0fa2b4900b9badd2e08fea09d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:14:00.067565 561 rebalancer_tool.cc:1074] tablet 9132d0f9d1ea4be49458a085c9e98a65: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:14:00.219110 561 rebalancer_tool.cc:1074] tablet dea30267166942258d47155fd0173713: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:00.321055 561 rebalancer_tool.cc:1074] tablet 49d89642d5364479877c7ba4700fdcee: 'a4cdf05256a0407c8929ac43eb55d44f' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:00.473371 561 rebalancer_tool.cc:1074] tablet d7ba7409f5bd4531bc5370019aca14c7: '793a0d5dd1634c5faaf0e1e4851e8294' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:00.729662 561 rebalancer_tool.cc:1074] tablet d9119ae1f5ca4c12ba7a4444edd210cf: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:14:00.814342 561 rebalancer_tool.cc:1074] tablet 30e4fb0a79814959aa25cf6eb34dfbb9: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:00.888648 561 rebalancer_tool.cc:1074] tablet 029be0389e244435bb08db2361729ad2: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:00.995152 561 rebalancer_tool.cc:1074] tablet 3905bce9ed6345f9979bda9500d1d053: '530893a961af4b95b6b687fa5ed22f9d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:14:01.122159 561 rebalancer_tool.cc:1074] tablet 3fb8c05985f34deebc145c3c51dc413d: 'fde134d0fa2b4900b9badd2e08fea09d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:14:18.883906 561 rebalancer_tool.cc:1136] tablet 9132d0f9d1ea4be49458a085c9e98a65: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:14:18.897255 561 rebalancer_tool.cc:1136] tablet 029be0389e244435bb08db2361729ad2: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:18.936486 561 rebalancer_tool.cc:1136] tablet 344eb942dc3d4188bc9baf3d1ecc4f7a: 'd0dbfb0e43f948c3922926aacc365fc1' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:14:18.945739 561 rebalancer_tool.cc:1136] tablet a3888675106d4d389718380627c95edf: '5388902973134958a030fd63ab4faba4' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:18.955490 561 rebalancer_tool.cc:1136] tablet d7ba7409f5bd4531bc5370019aca14c7: '793a0d5dd1634c5faaf0e1e4851e8294' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:14:18.972935 561 rebalancer_tool.cc:1136] tablet a6b4ef3ee1fd440384272cee496fc102: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:14:18.982985 561 rebalancer_tool.cc:1136] tablet 9a49047eedc54168af70a7b59fd7d770: 'fde134d0fa2b4900b9badd2e08fea09d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:14:18.993574 561 rebalancer_tool.cc:1136] tablet 016a1c41112441619700c662beee8c98: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:14:19.002180 561 rebalancer_tool.cc:1136] tablet cd84de9286eb4ad7b88b9f243eb2801c: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:14:19.011044 561 rebalancer_tool.cc:1136] tablet f4b9d916f3fb4a348a8702662c143435: 'a4cdf05256a0407c8929ac43eb55d44f' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:14:19.020154 561 rebalancer_tool.cc:1136] tablet 3fb8c05985f34deebc145c3c51dc413d: 'fde134d0fa2b4900b9badd2e08fea09d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:14:19.719774 561 rebalancer_tool.cc:1136] tablet 45da3dab7a204d378273a8985d91ace8: 'd0dbfb0e43f948c3922926aacc365fc1' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:14:19.730316 561 rebalancer_tool.cc:1136] tablet dea30267166942258d47155fd0173713: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:14:19.739454 561 rebalancer_tool.cc:1136] tablet 30e4fb0a79814959aa25cf6eb34dfbb9: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:14:19.751811 561 rebalancer_tool.cc:1136] tablet 49d89642d5364479877c7ba4700fdcee: 'a4cdf05256a0407c8929ac43eb55d44f' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:14:19.773689 561 rebalancer_tool.cc:1136] tablet d9119ae1f5ca4c12ba7a4444edd210cf: '530893a961af4b95b6b687fa5ed22f9d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:14:19.834899 561 rebalancer_tool.cc:1074] tablet 9088c0106bf249cab5b35c1171a7e7a0: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:19.903596 561 rebalancer_tool.cc:1074] tablet e57c283674c342169940a5903516ec93: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:19.982225 561 rebalancer_tool.cc:1074] tablet d7b192797b6c40d4853e6ae4583ef3ca: '6409961ee8c0445687c11724213c44a9' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:14:20.039770 561 rebalancer_tool.cc:1074] tablet 1bb01c0be7da4fe4b9c7c7160df7debc: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:20.134050 561 rebalancer_tool.cc:1074] tablet e571695c819e485483ece09278ee1e09: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:20.202373 561 rebalancer_tool.cc:1074] tablet 48266bf4ad9b4976b639553490ffebbe: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:20.294183 561 rebalancer_tool.cc:1074] tablet da7bb0a1956a452cbc1cbbb3999fd7e2: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:20.398780 561 rebalancer_tool.cc:1074] tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b: 'd0dbfb0e43f948c3922926aacc365fc1' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:20.556501 561 rebalancer_tool.cc:1074] tablet bb1ec320c4114eec8915b7f2bcffc2c3: '6409961ee8c0445687c11724213c44a9' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:20.729754 561 rebalancer_tool.cc:1074] tablet b96b54a566db4bd1a31e8fb30096c942: '530893a961af4b95b6b687fa5ed22f9d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:14:20.894088 561 rebalancer_tool.cc:1074] tablet 8377b1aa704c43b78f310e7440f6488e: 'fde134d0fa2b4900b9badd2e08fea09d' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:21.013893 561 rebalancer_tool.cc:1074] tablet 69d4d86cb68646bc9141c1e6df964561: '6409961ee8c0445687c11724213c44a9' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:21.108773 561 rebalancer_tool.cc:1074] tablet dcb14382755441afb4cc5c103425331c: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:21.185079 561 rebalancer_tool.cc:1074] tablet 1058a6f8db004b50a51a4c06152c7756: 'fde134d0fa2b4900b9badd2e08fea09d' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:14:21.309600 561 rebalancer_tool.cc:1074] tablet 81973d2082ab4bacb61722e51c146f30: '793a0d5dd1634c5faaf0e1e4851e8294' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:21.380954 561 rebalancer_tool.cc:1074] tablet 609db8c3f3b245eaa31c82915b61188f: '6409961ee8c0445687c11724213c44a9' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:21.470469 561 rebalancer_tool.cc:1074] tablet d58ff895fc8e4ddb84e2c1db165799d1: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:34.139317 561 rebalancer_tool.cc:1136] tablet 3905bce9ed6345f9979bda9500d1d053: '530893a961af4b95b6b687fa5ed22f9d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:14:34.152678 561 rebalancer_tool.cc:1136] tablet 9517b56a435b48e18c79d2a375e5a951: 'fde134d0fa2b4900b9badd2e08fea09d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:14:34.162282 561 rebalancer_tool.cc:1136] tablet 618ba33946424065a465217bae7cdbb0: 'fde134d0fa2b4900b9badd2e08fea09d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:14:34.208567 561 rebalancer_tool.cc:1074] tablet 8dd90464f4b84baabd90a44a122e28b1: 'fde134d0fa2b4900b9badd2e08fea09d' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:14:35.613631 561 rebalancer_tool.cc:1136] tablet 69d4d86cb68646bc9141c1e6df964561: '6409961ee8c0445687c11724213c44a9' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:14:35.629575 561 rebalancer_tool.cc:1136] tablet 1058a6f8db004b50a51a4c06152c7756: 'fde134d0fa2b4900b9badd2e08fea09d' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:14:35.645905 561 rebalancer_tool.cc:1136] tablet 8377b1aa704c43b78f310e7440f6488e: 'fde134d0fa2b4900b9badd2e08fea09d' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:14:35.662026 561 rebalancer_tool.cc:1136] tablet b96b54a566db4bd1a31e8fb30096c942: '530893a961af4b95b6b687fa5ed22f9d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:14:35.684602 561 rebalancer_tool.cc:1136] tablet dcb14382755441afb4cc5c103425331c: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:35.699771 561 rebalancer_tool.cc:1136] tablet bb1ec320c4114eec8915b7f2bcffc2c3: '6409961ee8c0445687c11724213c44a9' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:35.708561 561 rebalancer_tool.cc:1136] tablet d58ff895fc8e4ddb84e2c1db165799d1: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:35.718565 561 rebalancer_tool.cc:1136] tablet 2e1fbc1d1bb6439c9f8b64364ad9ab4b: 'd0dbfb0e43f948c3922926aacc365fc1' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:14:35.727506 561 rebalancer_tool.cc:1136] tablet da7bb0a1956a452cbc1cbbb3999fd7e2: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:14:35.737254 561 rebalancer_tool.cc:1136] tablet 48266bf4ad9b4976b639553490ffebbe: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:14:36.347391 561 rebalancer_tool.cc:1136] tablet 1bb01c0be7da4fe4b9c7c7160df7debc: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:14:36.366370 561 rebalancer_tool.cc:1136] tablet d7b192797b6c40d4853e6ae4583ef3ca: '6409961ee8c0445687c11724213c44a9' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:14:36.394100 561 rebalancer_tool.cc:1136] tablet e57c283674c342169940a5903516ec93: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:36.417963 561 rebalancer_tool.cc:1136] tablet e571695c819e485483ece09278ee1e09: '6bbc686598f24f518d7ca6bb2063058d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:36.429554 561 rebalancer_tool.cc:1136] tablet 9088c0106bf249cab5b35c1171a7e7a0: '5388902973134958a030fd63ab4faba4' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:14:36.491189 561 rebalancer_tool.cc:1074] tablet d0d2600fd84a4d839dc3366b08485e42: '6bbc686598f24f518d7ca6bb2063058d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:14:36.561058 561 rebalancer_tool.cc:1074] tablet a816d34a63b945b08eec771d9e7c2d55: 'a4cdf05256a0407c8929ac43eb55d44f' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:36.636956 561 rebalancer_tool.cc:1074] tablet 664cadf5232f4768adf6438e57f7b7c9: '6409961ee8c0445687c11724213c44a9' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:14:36.715292 561 rebalancer_tool.cc:1074] tablet 1f4fbfaadc474d59bfb23107f0342f7c: 'd0dbfb0e43f948c3922926aacc365fc1' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:36.774502 561 rebalancer_tool.cc:1074] tablet 38cf3aab9e1b4f28842ac16d02be2562: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:36.826480 561 rebalancer_tool.cc:1074] tablet ac08c6dba4dc4e90ab11a155ec6ecdfd: 'a4cdf05256a0407c8929ac43eb55d44f' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:36.881510 561 rebalancer_tool.cc:1074] tablet 0abfcd38ecc64acfac35c676b941581a: '6409961ee8c0445687c11724213c44a9' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:36.988322 561 rebalancer_tool.cc:1074] tablet 8baf804d83954d288576c313efe3d1cb: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:37.043623 561 rebalancer_tool.cc:1074] tablet f133ebe695a74f86be1a37ae661719ec: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:37.154592 561 rebalancer_tool.cc:1074] tablet e389fcad67cc4add9db2f34eb57745c8: 'fde134d0fa2b4900b9badd2e08fea09d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:37.446188 561 rebalancer_tool.cc:1074] tablet 5f25a3bcdef849499e4a02f01f4dbc0e: '6409961ee8c0445687c11724213c44a9' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:37.675882 561 rebalancer_tool.cc:1074] tablet c761d663c0bd48529edc9e454d5292ac: 'd0dbfb0e43f948c3922926aacc365fc1' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:14:37.781137 561 rebalancer_tool.cc:1074] tablet f7242c3728b84084a738fde7ae32ec0a: '793a0d5dd1634c5faaf0e1e4851e8294' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:37.834751 561 rebalancer_tool.cc:1074] tablet 96773dd7242d4223bdc663ee53bfdd3f: 'fde134d0fa2b4900b9badd2e08fea09d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:37.885269 561 rebalancer_tool.cc:1074] tablet cb88c79b4b1a40c2b45c2dd79ab894d4: '6409961ee8c0445687c11724213c44a9' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:38.038964 561 rebalancer_tool.cc:1074] tablet 3a82a00c33e1412f93ab071d5b0d824a: 'd0dbfb0e43f948c3922926aacc365fc1' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:38.136425 561 rebalancer_tool.cc:1074] tablet 2d0b457ba3564c52824f84b78cf14c23: '530893a961af4b95b6b687fa5ed22f9d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
W20240503 14:14:41.216872 6532 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.240.66:37039, user_credentials={real_user=slave}} blocked reactor thread for 63934.2us
I20240503 14:14:51.474695 561 rebalancer_tool.cc:1136] tablet 609db8c3f3b245eaa31c82915b61188f: '6409961ee8c0445687c11724213c44a9' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:14:51.485510 561 rebalancer_tool.cc:1136] tablet 81973d2082ab4bacb61722e51c146f30: '793a0d5dd1634c5faaf0e1e4851e8294' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:14:51.493870 561 rebalancer_tool.cc:1136] tablet 8dd90464f4b84baabd90a44a122e28b1: 'fde134d0fa2b4900b9badd2e08fea09d' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:14:51.565205 561 rebalancer_tool.cc:1074] tablet 710e09d5ab4e457fbbf54b3acaf5ff15: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:14:51.618755 561 rebalancer_tool.cc:1074] tablet 98d39010d3e24d9fbd33c559568f7330: 'd0dbfb0e43f948c3922926aacc365fc1' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:14:53.204949 561 rebalancer_tool.cc:1136] tablet 3a82a00c33e1412f93ab071d5b0d824a: 'd0dbfb0e43f948c3922926aacc365fc1' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:14:53.217382 561 rebalancer_tool.cc:1136] tablet 96773dd7242d4223bdc663ee53bfdd3f: 'fde134d0fa2b4900b9badd2e08fea09d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:54.577739 561 rebalancer_tool.cc:1136] tablet c761d663c0bd48529edc9e454d5292ac: 'd0dbfb0e43f948c3922926aacc365fc1' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:14:54.592483 561 rebalancer_tool.cc:1136] tablet 5f25a3bcdef849499e4a02f01f4dbc0e: '6409961ee8c0445687c11724213c44a9' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:14:54.601357 561 rebalancer_tool.cc:1136] tablet e389fcad67cc4add9db2f34eb57745c8: 'fde134d0fa2b4900b9badd2e08fea09d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:14:54.611008 561 rebalancer_tool.cc:1136] tablet f133ebe695a74f86be1a37ae661719ec: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:14:54.619313 561 rebalancer_tool.cc:1136] tablet 8baf804d83954d288576c313efe3d1cb: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:14:55.190804 561 rebalancer_tool.cc:1136] tablet ac08c6dba4dc4e90ab11a155ec6ecdfd: 'a4cdf05256a0407c8929ac43eb55d44f' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:14:55.211143 561 rebalancer_tool.cc:1136] tablet cb88c79b4b1a40c2b45c2dd79ab894d4: '6409961ee8c0445687c11724213c44a9' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
W20240503 14:14:55.686841 8445 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.240.70:39465, user_credentials={real_user=slave}} blocked reactor thread for 47816.6us
I20240503 14:14:55.864774 561 rebalancer_tool.cc:1136] tablet 1f4fbfaadc474d59bfb23107f0342f7c: 'd0dbfb0e43f948c3922926aacc365fc1' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:14:55.874080 561 rebalancer_tool.cc:1136] tablet 664cadf5232f4768adf6438e57f7b7c9: '6409961ee8c0445687c11724213c44a9' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:14:55.885464 561 rebalancer_tool.cc:1136] tablet a816d34a63b945b08eec771d9e7c2d55: 'a4cdf05256a0407c8929ac43eb55d44f' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:14:55.913278 561 rebalancer_tool.cc:1136] tablet d0d2600fd84a4d839dc3366b08485e42: '6bbc686598f24f518d7ca6bb2063058d' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:14:55.990554 561 rebalancer_tool.cc:1074] tablet bf316cde5631415faecbc1bb26690b79: '6bbc686598f24f518d7ca6bb2063058d' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:56.042439 561 rebalancer_tool.cc:1074] tablet 619951b364d44766a2f3064222ceb738: '530893a961af4b95b6b687fa5ed22f9d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:56.106706 561 rebalancer_tool.cc:1074] tablet ee914d743a2c46418f79728f3d32d1c5: 'd0dbfb0e43f948c3922926aacc365fc1' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:56.157366 561 rebalancer_tool.cc:1074] tablet 0225203f2de2498eb448e066ce545a64: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:56.243824 561 rebalancer_tool.cc:1074] tablet 213bc70b5cf14440ab88c754df0b8f7f: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:56.315531 561 rebalancer_tool.cc:1074] tablet 5be4220923ed4c5388a7a15755cb685d: '530893a961af4b95b6b687fa5ed22f9d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:14:56.403863 561 rebalancer_tool.cc:1074] tablet 823dde634aac449ea8cd21d38ecfb22b: '793a0d5dd1634c5faaf0e1e4851e8294' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:14:56.486413 561 rebalancer_tool.cc:1074] tablet dc9c29880eea4cd9a11eb24eeb3bd228: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:56.780532 561 rebalancer_tool.cc:1074] tablet 8868e8d0fd6b4a14970df92960ca0b80: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:56.903679 561 rebalancer_tool.cc:1074] tablet 859456c340dc47bf8fc600ebe8c4b984: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:14:57.054173 561 rebalancer_tool.cc:1074] tablet d2bb66488d714d39900331f4a257e8b3: 'd0dbfb0e43f948c3922926aacc365fc1' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:14:57.186026 561 rebalancer_tool.cc:1074] tablet 519fed0fc03b4b9788803aeb0bbcfdad: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:14:57.277436 561 rebalancer_tool.cc:1074] tablet edd300244be74e3aa94d7ae35deeb470: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move scheduled
I20240503 14:14:57.382710 561 rebalancer_tool.cc:1074] tablet 25eb1d6c445940d6bd8753a88381bb8c: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
W20240503 14:14:58.724931 8997 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.20.240.68:38371, user_credentials={real_user=slave}} blocked reactor thread for 85167.1us
W20240503 14:15:00.896095 9222 meta_cache.cc:1261] Time spent looking up entry by key: real 0.051s user 0.000s sys 0.000s
I20240503 14:15:08.756774 561 rebalancer_tool.cc:1136] tablet 2d0b457ba3564c52824f84b78cf14c23: '530893a961af4b95b6b687fa5ed22f9d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:15:08.768687 561 rebalancer_tool.cc:1136] tablet 710e09d5ab4e457fbbf54b3acaf5ff15: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:15:08.778726 561 rebalancer_tool.cc:1136] tablet 98d39010d3e24d9fbd33c559568f7330: 'd0dbfb0e43f948c3922926aacc365fc1' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:15:08.788547 561 rebalancer_tool.cc:1136] tablet f7242c3728b84084a738fde7ae32ec0a: '793a0d5dd1634c5faaf0e1e4851e8294' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:15:09.589792 561 rebalancer_tool.cc:1136] tablet 0abfcd38ecc64acfac35c676b941581a: '6409961ee8c0445687c11724213c44a9' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:15:09.694638 561 rebalancer_tool.cc:1136] tablet 38cf3aab9e1b4f28842ac16d02be2562: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:15:09.852669 561 rebalancer_tool.cc:1074] tablet f7bdba20d1844eefb6d3b29c5deb2873: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:15:09.952504 561 rebalancer_tool.cc:1074] tablet 5815f4c96eef4130aa0582d33a24b391: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:15:10.006481 561 rebalancer_tool.cc:1074] tablet f11b07607b0e430db646172f863cdc1f: 'd0dbfb0e43f948c3922926aacc365fc1' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:15:10.060215 561 rebalancer_tool.cc:1074] tablet 88f88e7eb7434450af76714256729887: 'a4cdf05256a0407c8929ac43eb55d44f' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:15:10.134232 561 rebalancer_tool.cc:1074] tablet 92f507cdcfb04eb4a9eb448773206783: '6bbc686598f24f518d7ca6bb2063058d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:15:16.000216 561 rebalancer_tool.cc:1136] tablet 519fed0fc03b4b9788803aeb0bbcfdad: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:15:16.013185 561 rebalancer_tool.cc:1136] tablet 859456c340dc47bf8fc600ebe8c4b984: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:15:16.765543 561 rebalancer_tool.cc:1136] tablet 8868e8d0fd6b4a14970df92960ca0b80: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:15:16.775362 561 rebalancer_tool.cc:1136] tablet dc9c29880eea4cd9a11eb24eeb3bd228: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:15:16.783434 561 rebalancer_tool.cc:1136] tablet 823dde634aac449ea8cd21d38ecfb22b: '793a0d5dd1634c5faaf0e1e4851e8294' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:15:16.792289 561 rebalancer_tool.cc:1136] tablet 5be4220923ed4c5388a7a15755cb685d: '530893a961af4b95b6b687fa5ed22f9d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:15:16.801272 561 rebalancer_tool.cc:1136] tablet 0225203f2de2498eb448e066ce545a64: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move completed: OK
I20240503 14:15:16.809593 561 rebalancer_tool.cc:1136] tablet d2bb66488d714d39900331f4a257e8b3: 'd0dbfb0e43f948c3922926aacc365fc1' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:15:16.817816 561 rebalancer_tool.cc:1136] tablet ee914d743a2c46418f79728f3d32d1c5: 'd0dbfb0e43f948c3922926aacc365fc1' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:15:16.827078 561 rebalancer_tool.cc:1136] tablet 619951b364d44766a2f3064222ceb738: '530893a961af4b95b6b687fa5ed22f9d' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:15:16.837814 561 rebalancer_tool.cc:1136] tablet edd300244be74e3aa94d7ae35deeb470: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:15:16.848006 561 rebalancer_tool.cc:1136] tablet bf316cde5631415faecbc1bb26690b79: '6bbc686598f24f518d7ca6bb2063058d' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:15:16.856969 561 rebalancer_tool.cc:1136] tablet 213bc70b5cf14440ab88c754df0b8f7f: '6409961ee8c0445687c11724213c44a9' -> 'fde134d0fa2b4900b9badd2e08fea09d' move completed: OK
I20240503 14:15:16.930696 561 rebalancer_tool.cc:1074] tablet 1af2fb3fb3fd4dfe873d124232d812b5: 'fde134d0fa2b4900b9badd2e08fea09d' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:15:17.021788 561 rebalancer_tool.cc:1074] tablet f6151ea340334090a5c0e79ca65af5c1: '6bbc686598f24f518d7ca6bb2063058d' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:15:17.090903 561 rebalancer_tool.cc:1074] tablet 4d4c447b816b430caf4b213f896e51e3: 'fde134d0fa2b4900b9badd2e08fea09d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:15:17.177845 561 rebalancer_tool.cc:1074] tablet 72c7ff14d43342439f935389b03ecad0: 'd0dbfb0e43f948c3922926aacc365fc1' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:15:17.237502 561 rebalancer_tool.cc:1074] tablet 529c250699cb4e09bae6129d138e37c2: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:15:17.322831 561 rebalancer_tool.cc:1074] tablet 05a67063845044eea2d56c34cdfd4ab1: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:15:17.386117 561 rebalancer_tool.cc:1074] tablet fbd2cac7ae8e45b6bd407b119efe3a51: 'fde134d0fa2b4900b9badd2e08fea09d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:15:17.470701 561 rebalancer_tool.cc:1074] tablet 06e418c4c2cf46608ba454d48fae9285: 'd0dbfb0e43f948c3922926aacc365fc1' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:15:17.551851 561 rebalancer_tool.cc:1074] tablet 130a2a0b5aa344d3be535cfde8990d39: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:15:17.730405 561 rebalancer_tool.cc:1074] tablet ff8d8c22d99c462b931a77297c5ff1ff: '6bbc686598f24f518d7ca6bb2063058d' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:15:18.022727 561 rebalancer_tool.cc:1074] tablet 1b4544aefcc54a828dc3b75726367f70: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:15:29.583436 561 rebalancer_tool.cc:1136] tablet f11b07607b0e430db646172f863cdc1f: 'd0dbfb0e43f948c3922926aacc365fc1' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:15:30.319368 561 rebalancer_tool.cc:1136] tablet 25eb1d6c445940d6bd8753a88381bb8c: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:15:30.328877 561 rebalancer_tool.cc:1136] tablet 5815f4c96eef4130aa0582d33a24b391: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:15:30.388803 561 rebalancer_tool.cc:1074] tablet 917de496d0cb4c6d94c6dcf630b90f4a: 'd0dbfb0e43f948c3922926aacc365fc1' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:15:30.458494 561 rebalancer_tool.cc:1074] tablet 02559d1ab4824298ba76a44a7f05303b: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move scheduled
I20240503 14:15:30.541203 561 rebalancer_tool.cc:1074] tablet a82470f287274b1c878089fa9d56ef40: '6bbc686598f24f518d7ca6bb2063058d' -> '530893a961af4b95b6b687fa5ed22f9d' move scheduled
I20240503 14:15:33.346014 561 rebalancer_tool.cc:1136] tablet ff8d8c22d99c462b931a77297c5ff1ff: '6bbc686598f24f518d7ca6bb2063058d' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:15:33.355026 561 rebalancer_tool.cc:1136] tablet 130a2a0b5aa344d3be535cfde8990d39: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:15:33.364025 561 rebalancer_tool.cc:1136] tablet 06e418c4c2cf46608ba454d48fae9285: 'd0dbfb0e43f948c3922926aacc365fc1' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:15:33.375880 561 rebalancer_tool.cc:1136] tablet fbd2cac7ae8e45b6bd407b119efe3a51: 'fde134d0fa2b4900b9badd2e08fea09d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:15:33.397547 561 rebalancer_tool.cc:1136] tablet 529c250699cb4e09bae6129d138e37c2: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:15:33.422428 561 rebalancer_tool.cc:1136] tablet 72c7ff14d43342439f935389b03ecad0: 'd0dbfb0e43f948c3922926aacc365fc1' -> '793a0d5dd1634c5faaf0e1e4851e8294' move completed: OK
I20240503 14:15:33.432829 561 rebalancer_tool.cc:1136] tablet 4d4c447b816b430caf4b213f896e51e3: 'fde134d0fa2b4900b9badd2e08fea09d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:15:33.442103 561 rebalancer_tool.cc:1136] tablet f6151ea340334090a5c0e79ca65af5c1: '6bbc686598f24f518d7ca6bb2063058d' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:15:34.336292 561 rebalancer_tool.cc:1136] tablet 92f507cdcfb04eb4a9eb448773206783: '6bbc686598f24f518d7ca6bb2063058d' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:15:34.347446 561 rebalancer_tool.cc:1136] tablet 88f88e7eb7434450af76714256729887: 'a4cdf05256a0407c8929ac43eb55d44f' -> 'd0dbfb0e43f948c3922926aacc365fc1' move completed: OK
I20240503 14:15:35.064651 561 rebalancer_tool.cc:1136] tablet f7bdba20d1844eefb6d3b29c5deb2873: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6409961ee8c0445687c11724213c44a9' move completed: OK
I20240503 14:15:35.136828 561 rebalancer_tool.cc:1074] tablet a88d8b316d064c4e8fdccaba77d656c2: '793a0d5dd1634c5faaf0e1e4851e8294' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:15:35.186934 561 rebalancer_tool.cc:1074] tablet 7f807f1ac8d8417aaffd9254993b2085: 'd0dbfb0e43f948c3922926aacc365fc1' -> 'a4cdf05256a0407c8929ac43eb55d44f' move scheduled
I20240503 14:15:35.259543 561 rebalancer_tool.cc:1074] tablet 4b6b6e0cac9a4ea6a0189970d88f9d72: 'a4cdf05256a0407c8929ac43eb55d44f' -> '793a0d5dd1634c5faaf0e1e4851e8294' move scheduled
I20240503 14:15:35.325815 561 rebalancer_tool.cc:1074] tablet 361b2d07d8b44d128acfb3cc941ac01a: '6bbc686598f24f518d7ca6bb2063058d' -> '6409961ee8c0445687c11724213c44a9' move scheduled
I20240503 14:15:35.388846 561 rebalancer_tool.cc:1074] tablet 133f5691d69b461f8e7502ecffdb8a89: 'a4cdf05256a0407c8929ac43eb55d44f' -> 'd0dbfb0e43f948c3922926aacc365fc1' move scheduled
I20240503 14:15:35.457895 561 rebalancer_tool.cc:1074] tablet 027c88573444465a9ebcaa9589728e74: '530893a961af4b95b6b687fa5ed22f9d' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
I20240503 14:15:35.521322 561 rebalancer_tool.cc:1074] tablet bca56ccf82a74df7bd8006e31e02bd20: '530893a961af4b95b6b687fa5ed22f9d' -> '6bbc686598f24f518d7ca6bb2063058d' move scheduled
W20240503 14:15:36.657999 13741 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:15:36.327061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.240.70:39465 (local address 127.0.0.1:53190)
0503 14:15:36.328390 (+ 1329us) negotiation.cc:107] Waiting for socket to connect
0503 14:15:36.328405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0503 14:15:36.328488 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:15:36.656586 (+328098us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":19632,"threads_started":1}
I20240503 14:15:41.135360 561 rebalancer_tool.cc:1136] tablet 02559d1ab4824298ba76a44a7f05303b: '530893a961af4b95b6b687fa5ed22f9d' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:15:41.145035 561 rebalancer_tool.cc:1136] tablet 917de496d0cb4c6d94c6dcf630b90f4a: 'd0dbfb0e43f948c3922926aacc365fc1' -> 'a4cdf05256a0407c8929ac43eb55d44f' move completed: OK
I20240503 14:15:41.153669 561 rebalancer_tool.cc:1136] tablet 1b4544aefcc54a828dc3b75726367f70: 'a4cdf05256a0407c8929ac43eb55d44f' -> '530893a961af4b95b6b687fa5ed22f9d' move completed: OK
I20240503 14:15:41.162652 561 rebalancer_tool.cc:1136] tablet 1af2fb3fb3fd4dfe873d124232d812b5: 'fde134d0fa2b4900b9badd2e08fea09d' -> '5388902973134958a030fd63ab4faba4' move completed: OK
I20240503 14:15:41.166077 561 rebalancer_tool.cc:1123] tablet 05a67063845044eea2d56c34cdfd4ab1: '5388902973134958a030fd63ab4faba4' -> '6bbc686598f24f518d7ca6bb2063058d' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:41.169945 561 rebalancer_tool.cc:1084] tablet 3cf751c4f28d4358b4f47c3984cfd376: 'a4cdf05256a0407c8929ac43eb55d44f' -> '793a0d5dd1634c5faaf0e1e4851e8294' move ignored: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:41.170086 561 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): unacceptable health status UNAVAILABLE
I20240503 14:15:42.399488 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 27961
W20240503 14:15:42.399919 30746 consensus_peers.cc:485] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:42.408354 28083 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:15:42.513813 31008 consensus_peers.cc:485] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:42.579195 13876 raft_consensus.cc:482] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bbc686598f24f518d7ca6bb2063058d)
I20240503 14:15:42.579330 13876 raft_consensus.cc:504] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:42.579965 13876 leader_election.cc:288] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:42.581413 28110 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:42.581992 28109 leader_election.cc:334] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:42.582115 28109 leader_election.cc:302] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:42.582379 13876 raft_consensus.cc:2726] T 05a67063845044eea2d56c34cdfd4ab1 P 6409961ee8c0445687c11724213c44a9 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:42.643981 30746 consensus_peers.cc:485] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.695770 28369 consensus_peers.cc:485] T 4e5126a38fc740d49f638d1b3fb8a9c9 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.741776 28370 consensus_peers.cc:485] T d7b192797b6c40d4853e6ae4583ef3ca P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.746263 31138 consensus_peers.cc:485] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.752425 28370 consensus_peers.cc:485] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.770828 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:42.810928 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:42.830929 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 27961
W20240503 14:15:42.835247 30745 consensus_peers.cc:485] T 9088c0106bf249cab5b35c1171a7e7a0 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.836371 28110 consensus_peers.cc:485] T 1bb01c0be7da4fe4b9c7c7160df7debc P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:42.838222 31138 consensus_peers.cc:485] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:42.885505 13814 raft_consensus.cc:482] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a4cdf05256a0407c8929ac43eb55d44f)
I20240503 14:15:42.885715 13814 raft_consensus.cc:504] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:42.886498 13814 leader_election.cc:288] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
W20240503 14:15:42.887881 28110 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:42.888301 28109 leader_election.cc:334] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:42.888409 28109 leader_election.cc:302] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6409961ee8c0445687c11724213c44a9; no voters: 6bbc686598f24f518d7ca6bb2063058d, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:42.888638 13814 raft_consensus.cc:2726] T 519fed0fc03b4b9788803aeb0bbcfdad P 6409961ee8c0445687c11724213c44a9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:42.891255 31008 connection.cc:535] client connection to 127.20.240.65:33293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:42.891779 30746 connection.cc:535] client connection to 127.20.240.65:33293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:42.892102 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 28092
W20240503 14:15:42.892433 28370 connection.cc:535] client connection to 127.20.240.65:33293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:42.892551 31139 connection.cc:535] client connection to 127.20.240.65:33293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:42.892541 31008 consensus_peers.cc:485] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.892820 28110 connection.cc:535] client connection to 127.20.240.65:33293 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:42.894156 31139 consensus_peers.cc:485] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.894166 28369 connection.cc:535] server connection from 127.20.240.65:55553 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:42.894611 31139 consensus_peers.cc:485] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.894778 31139 consensus_peers.cc:485] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.894881 30746 consensus_peers.cc:485] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895005 30746 consensus_peers.cc:485] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895118 30746 consensus_peers.cc:485] T 611bac13cf32492b87c2f57752c59462 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895388 28110 consensus_peers.cc:485] T da7bb0a1956a452cbc1cbbb3999fd7e2 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895567 28110 consensus_peers.cc:485] T d0d2600fd84a4d839dc3366b08485e42 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895727 28110 consensus_peers.cc:485] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895874 28110 consensus_peers.cc:485] T a816d34a63b945b08eec771d9e7c2d55 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895859 28370 consensus_peers.cc:485] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.895980 28370 consensus_peers.cc:485] T 7b82b7a5fb7c4e13b126f6d47164a463 P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896090 28370 consensus_peers.cc:485] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896026 28110 consensus_peers.cc:485] T 0225203f2de2498eb448e066ce545a64 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896190 28370 consensus_peers.cc:485] T b96b54a566db4bd1a31e8fb30096c942 P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896279 28370 consensus_peers.cc:485] T 84dbd48ddf494ae2b9e23ba2b294376c P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896399 28370 consensus_peers.cc:485] T 400ce95594a743108bb4ee1e64458956 P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896432 28110 consensus_peers.cc:485] T 618ba33946424065a465217bae7cdbb0 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896499 28370 consensus_peers.cc:485] T 8baf804d83954d288576c313efe3d1cb P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896605 28370 consensus_peers.cc:485] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896641 28110 consensus_peers.cc:485] T 8ddd3f269b35481baa313da70ee987de P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896826 28110 consensus_peers.cc:485] T f11b07607b0e430db646172f863cdc1f P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.896957 28110 consensus_peers.cc:485] T 3905bce9ed6345f9979bda9500d1d053 P 6409961ee8c0445687c11724213c44a9 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.902957 31138 consensus_peers.cc:485] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:42.913416 28213 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:15:42.928575 30746 consensus_peers.cc:485] T 5be4220923ed4c5388a7a15755cb685d P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:42.934993 31008 consensus_peers.cc:485] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.946422 30746 consensus_peers.cc:485] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Couldn't send request to peer 793a0d5dd1634c5faaf0e1e4851e8294. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:42.978780 31007 consensus_peers.cc:485] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:43.039675 31008 consensus_peers.cc:485] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.064895 28369 consensus_peers.cc:485] T ee914d743a2c46418f79728f3d32d1c5 P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:43.096284 30746 consensus_peers.cc:485] T 5e1c93fac26f4e0092c5806fe1aef59b P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.126811 31008 consensus_peers.cc:485] T d4c83d087d9c457db809bb3fabecf260 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.210820 28370 consensus_peers.cc:485] T d7ba7409f5bd4531bc5370019aca14c7 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.221079 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:43.245949 31139 consensus_peers.cc:485] T e389fcad67cc4add9db2f34eb57745c8 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.270610 30745 consensus_peers.cc:485] T 027c88573444465a9ebcaa9589728e74 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.281159 28370 consensus_peers.cc:485] T 029be0389e244435bb08db2361729ad2 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.285718 31008 consensus_peers.cc:485] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:43.294677 14307 raft_consensus.cc:482] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.294870 14307 raft_consensus.cc:504] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:43.295526 14307 leader_election.cc:288] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:43.298164 31139 leader_election.cc:334] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.298377 31139 leader_election.cc:334] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:43.298491 31139 leader_election.cc:302] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:43.298954 14307 raft_consensus.cc:2726] T e1e4fa6d5c704d7f9bd43767b4f77e5b P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.307548 28109 consensus_peers.cc:485] T 016a1c41112441619700c662beee8c98 P 6409961ee8c0445687c11724213c44a9 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:43.308758 28370 consensus_peers.cc:485] T 664cadf5232f4768adf6438e57f7b7c9 P 5388902973134958a030fd63ab4faba4 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.320246 28110 consensus_peers.cc:485] T cd84de9286eb4ad7b88b9f243eb2801c P 6409961ee8c0445687c11724213c44a9 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:15:43.321954 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 28092
W20240503 14:15:43.331568 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:43.366742 14358 raft_consensus.cc:482] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.366955 14358 raft_consensus.cc:504] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:43.367571 14358 leader_election.cc:288] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:43.368932 30746 leader_election.cc:334] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.382555 28369 consensus_peers.cc:485] T 1f4fbfaadc474d59bfb23107f0342f7c P 5388902973134958a030fd63ab4faba4 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:15:43.392442 30744 connection.cc:535] client connection to 127.20.240.66:37039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:43.393206 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 28352
W20240503 14:15:43.393893 30744 leader_election.cc:334] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111)
I20240503 14:15:43.394030 30744 leader_election.cc:302] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294
W20240503 14:15:43.394718 30743 connection.cc:535] server connection from 127.20.240.66:46535 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:43.394737 14389 raft_consensus.cc:2726] T fbd2cac7ae8e45b6bd407b119efe3a51 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.395969 28368 connection.cc:535] client connection to 127.20.240.66:37039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:43.397415 28368 consensus_peers.cc:485] T 130a2a0b5aa344d3be535cfde8990d39 P 5388902973134958a030fd63ab4faba4 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.397558 28368 consensus_peers.cc:485] T 49d89642d5364479877c7ba4700fdcee P 5388902973134958a030fd63ab4faba4 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.397655 28368 consensus_peers.cc:485] T 9a49047eedc54168af70a7b59fd7d770 P 5388902973134958a030fd63ab4faba4 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.397753 28368 consensus_peers.cc:485] T ac08c6dba4dc4e90ab11a155ec6ecdfd P 5388902973134958a030fd63ab4faba4 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.397840 28368 consensus_peers.cc:485] T 710e09d5ab4e457fbbf54b3acaf5ff15 P 5388902973134958a030fd63ab4faba4 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.400100 31006 consensus_peers.cc:485] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.401052 28473 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:15:43.411310 31006 consensus_peers.cc:485] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.417913 14354 raft_consensus.cc:482] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.418092 14354 raft_consensus.cc:504] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:43.418725 14354 leader_election.cc:288] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:43.420388 31007 leader_election.cc:334] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:43.420964 31008 leader_election.cc:334] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.421087 31008 leader_election.cc:302] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.421351 14354 raft_consensus.cc:2726] T 6b263006614c46b4ac0648efb7a37a8f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.436517 31139 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.452124 31008 consensus_peers.cc:485] T f7e304f721e447bc8c4cc6eec3db872b P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.454198 31138 consensus_peers.cc:485] T 68ab254ac309475984065af1a01b537f P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.456537 30744 consensus_peers.cc:485] T dea30267166942258d47155fd0173713 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.467021 30746 consensus_peers.cc:485] T e38e89746d9944e5a12f6b68a0ee13db P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.479982 31138 consensus_peers.cc:485] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.509927 31006 consensus_peers.cc:485] T c761d663c0bd48529edc9e454d5292ac P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.518890 31137 consensus_peers.cc:485] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.519176 30746 consensus_peers.cc:485] T 8c2df0906f1e49e2a36b4e4ad1dbc418 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:43.531281 14390 raft_consensus.cc:482] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.531486 14390 raft_consensus.cc:504] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:43.532223 14390 leader_election.cc:288] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
W20240503 14:15:43.533380 31139 leader_election.cc:334] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.543352 31007 consensus_peers.cc:485] T 2e1fbc1d1bb6439c9f8b64364ad9ab4b P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.572885 30744 consensus_peers.cc:485] T 215498be790445b3948b1c5e2823cec4 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.573514 14390 raft_consensus.cc:482] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.573673 14390 raft_consensus.cc:504] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:43.574442 14390 leader_election.cc:288] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:43.574676 14354 raft_consensus.cc:482] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
W20240503 14:15:43.575008 30744 consensus_peers.cc:485] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.575119 14354 raft_consensus.cc:504] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
W20240503 14:15:43.575856 31139 leader_election.cc:334] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.575992 14354 leader_election.cc:288] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689)
I20240503 14:15:43.576428 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd90464f4b84baabd90a44a122e28b1" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
W20240503 14:15:43.576956 31008 leader_election.cc:334] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.577029 31136 leader_election.cc:302] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:15:43.577193 30808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2e220a0034a9aac672f6ccdf497d1" candidate_uuid: "fde134d0fa2b4900b9badd2e08fea09d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d0dbfb0e43f948c3922926aacc365fc1" is_pre_election: true
I20240503 14:15:43.577309 14390 raft_consensus.cc:2726] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:43.577697 31005 leader_election.cc:302] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294, d0dbfb0e43f948c3922926aacc365fc1
I20240503 14:15:43.577951 14354 raft_consensus.cc:2726] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.579969 31006 consensus_peers.cc:485] T 9132d0f9d1ea4be49458a085c9e98a65 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.586422 31138 consensus_peers.cc:485] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:43.595263 14390 raft_consensus.cc:482] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.595394 14390 raft_consensus.cc:504] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } }
I20240503 14:15:43.595901 14390 leader_election.cc:288] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:43.597246 31139 leader_election.cc:334] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.597405 31139 leader_election.cc:334] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:43.597484 31139 leader_election.cc:302] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:43.597692 14390 raft_consensus.cc:2726] T 06e418c4c2cf46608ba454d48fae9285 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:43.604858 14390 raft_consensus.cc:482] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.605027 14390 raft_consensus.cc:504] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:43.605655 14390 leader_election.cc:288] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:43.607031 31139 leader_election.cc:334] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.607177 31137 leader_election.cc:334] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111)
I20240503 14:15:43.607304 31137 leader_election.cc:302] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.607529 14390 raft_consensus.cc:2726] T 529c250699cb4e09bae6129d138e37c2 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.636161 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:43.638813 31138 consensus_peers.cc:485] T b7b96e9a7f314f84a38fee177ade3da2 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:43.643653 14390 raft_consensus.cc:482] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.643841 14390 raft_consensus.cc:504] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:43.644510 14390 leader_election.cc:288] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:43.645882 31139 leader_election.cc:334] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.647898 30746 consensus_peers.cc:485] T f6151ea340334090a5c0e79ca65af5c1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:43.652462 14358 raft_consensus.cc:482] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.652652 14358 raft_consensus.cc:504] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:43.653298 14358 leader_election.cc:288] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:43.654860 30745 leader_election.cc:334] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:43.656147 30746 leader_election.cc:334] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.656292 30746 leader_election.cc:302] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.656531 14358 raft_consensus.cc:2726] T 98a0a9b88c6a4d8ba55e2aa4662dcdbe P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:43.671643 31008 consensus_peers.cc:485] T 917de496d0cb4c6d94c6dcf630b90f4a P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.682641 31137 consensus_peers.cc:485] T 361b2d07d8b44d128acfb3cc941ac01a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.685981 31137 consensus_peers.cc:485] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.722690 13825 raft_consensus.cc:482] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.722914 13825 raft_consensus.cc:504] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:43.723562 13825 leader_election.cc:288] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:43.725091 31008 leader_election.cc:334] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:43.725291 31008 leader_election.cc:334] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.725428 31008 leader_election.cc:302] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:43.725692 13825 raft_consensus.cc:2726] T 8868e8d0fd6b4a14970df92960ca0b80 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:43.727911 30746 consensus_peers.cc:485] T 96773dd7242d4223bdc663ee53bfdd3f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:43.742919 14389 raft_consensus.cc:482] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.743101 14389 raft_consensus.cc:504] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:43.743739 14389 leader_election.cc:288] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:43.744182 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd90464f4b84baabd90a44a122e28b1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:43.744417 31200 raft_consensus.cc:2454] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
I20240503 14:15:43.744917 30744 leader_election.cc:302] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters:
W20240503 14:15:43.745198 30746 leader_election.cc:334] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.745220 14389 raft_consensus.cc:2781] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:43.745334 14389 raft_consensus.cc:482] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.745436 14389 raft_consensus.cc:3037] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:43.748181 14389 raft_consensus.cc:504] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
W20240503 14:15:43.748714 31006 consensus_peers.cc:485] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.748756 14389 leader_election.cc:288] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:43.749244 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8dd90464f4b84baabd90a44a122e28b1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d"
I20240503 14:15:43.749491 31200 raft_consensus.cc:3037] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:43.750399 30746 leader_election.cc:334] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.751546 31200 raft_consensus.cc:2454] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 2.
I20240503 14:15:43.751899 30744 leader_election.cc:302] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d, d0dbfb0e43f948c3922926aacc365fc1; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.752135 14389 raft_consensus.cc:2781] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:43.752271 14389 raft_consensus.cc:686] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:15:43.752550 14389 consensus_queue.cc:237] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:43.756717 27899 catalog_manager.cc:5484] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 1 to 2, leader changed from 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 2 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } }
W20240503 14:15:43.757098 31006 consensus_peers.cc:485] T e571695c819e485483ece09278ee1e09 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.799146 13530 raft_consensus.cc:482] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.799388 13530 raft_consensus.cc:504] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:43.800212 13530 leader_election.cc:288] T ff8d8c22d99c462b931a77297c5ff1ff P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 530893a961af4b95b6b687fa5ed22f9d (127.20.240.72:41331)
I20240503 14:15:43.816653 14403 raft_consensus.cc:482] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.817067 14403 raft_consensus.cc:504] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:43.818138 14403 leader_election.cc:288] T 5815f4c96eef4130aa0582d33a24b391 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:43.818879 14390 raft_consensus.cc:482] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.819051 14390 raft_consensus.cc:504] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:43.819893 14390 leader_election.cc:288] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:43.820144 14390 raft_consensus.cc:482] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.820297 14390 raft_consensus.cc:504] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:43.820811 14390 leader_election.cc:288] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:43.823588 31008 consensus_peers.cc:485] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.825834 31138 leader_election.cc:334] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:43.827160 31139 leader_election.cc:334] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.827322 31139 leader_election.cc:302] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6bbc686598f24f518d7ca6bb2063058d, 793a0d5dd1634c5faaf0e1e4851e8294
W20240503 14:15:43.827713 31139 leader_election.cc:334] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.828137 14402 raft_consensus.cc:482] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.828397 14402 raft_consensus.cc:504] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } }
I20240503 14:15:43.829058 14390 raft_consensus.cc:2726] T 72c7ff14d43342439f935389b03ecad0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:15:43.830570 31200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff8d8c22d99c462b931a77297c5ff1ff" candidate_uuid: "5388902973134958a030fd63ab4faba4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "530893a961af4b95b6b687fa5ed22f9d" is_pre_election: true
I20240503 14:15:43.830780 31200 raft_consensus.cc:2454] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5388902973134958a030fd63ab4faba4 in term 1.
I20240503 14:15:43.832768 14402 leader_election.cc:288] T fba8583c4254431e8f95633c4998fc88 P 5388902973134958a030fd63ab4faba4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
I20240503 14:15:43.836251 28430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25eb1d6c445940d6bd8753a88381bb8c" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
I20240503 14:15:43.837306 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 28352
I20240503 14:15:43.837253 28430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "859456c340dc47bf8fc600ebe8c4b984" candidate_uuid: "530893a961af4b95b6b687fa5ed22f9d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5388902973134958a030fd63ab4faba4" is_pre_election: true
W20240503 14:15:43.840430 31137 consensus_peers.cc:485] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.841017 30744 consensus_peers.cc:485] T d9119ae1f5ca4c12ba7a4444edd210cf P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Couldn't send request to peer 6409961ee8c0445687c11724213c44a9. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.857787 14358 raft_consensus.cc:482] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.857951 14358 raft_consensus.cc:504] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:43.858551 14358 leader_election.cc:288] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:43.859169 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2e220a0034a9aac672f6ccdf497d1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d" is_pre_election: true
I20240503 14:15:43.859419 31070 raft_consensus.cc:2454] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 1.
W20240503 14:15:43.859812 30746 leader_election.cc:334] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:43.859977 30745 leader_election.cc:302] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.860265 14358 raft_consensus.cc:2781] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:15:43.860386 14358 raft_consensus.cc:482] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.860509 14358 raft_consensus.cc:3037] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:15:43.863749 14358 raft_consensus.cc:504] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:43.864514 14358 leader_election.cc:288] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Requested vote from peers 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293), fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707)
I20240503 14:15:43.865197 31070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a2e220a0034a9aac672f6ccdf497d1" candidate_uuid: "d0dbfb0e43f948c3922926aacc365fc1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fde134d0fa2b4900b9badd2e08fea09d"
I20240503 14:15:43.865413 31070 raft_consensus.cc:3037] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:15:43.865937 30746 leader_election.cc:334] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.866634 30746 consensus_peers.cc:485] T d58ff895fc8e4ddb84e2c1db165799d1 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:43.867748 31070 raft_consensus.cc:2454] T 36a2e220a0034a9aac672f6ccdf497d1 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dbfb0e43f948c3922926aacc365fc1 in term 2.
I20240503 14:15:43.868113 30745 leader_election.cc:302] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1, fde134d0fa2b4900b9badd2e08fea09d; no voters: 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.868359 14389 raft_consensus.cc:2781] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:15:43.868533 14389 raft_consensus.cc:686] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 LEADER]: Becoming Leader. State: Replica: d0dbfb0e43f948c3922926aacc365fc1, State: Running, Role: LEADER
I20240503 14:15:43.868857 14389 consensus_queue.cc:237] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:43.874065 27899 catalog_manager.cc:5484] T 36a2e220a0034a9aac672f6ccdf497d1 P d0dbfb0e43f948c3922926aacc365fc1 reported cstate change: term changed from 1 to 2, leader changed from 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65) to d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69). New cstate: current_term: 2 leader_uuid: "d0dbfb0e43f948c3922926aacc365fc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } }
I20240503 14:15:43.879528 14389 raft_consensus.cc:482] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 793a0d5dd1634c5faaf0e1e4851e8294)
I20240503 14:15:43.879688 14389 raft_consensus.cc:504] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } }
I20240503 14:15:43.880275 14389 leader_election.cc:288] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:43.882138 30746 leader_election.cc:334] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:43.890077 30744 leader_election.cc:334] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111)
I20240503 14:15:43.890204 30744 leader_election.cc:302] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.890522 14389 raft_consensus.cc:2726] T 88f88e7eb7434450af76714256729887 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:43.896461 30745 consensus_peers.cc:485] T 4d4c447b816b430caf4b213f896e51e3 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.904796 30746 connection.cc:535] client connection to 127.20.240.68:38371 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:43.904796 31007 connection.cc:535] server connection from 127.20.240.68:50449 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:43.904867 31137 connection.cc:535] server connection from 127.20.240.68:40129 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:43.905156 27888 connection.cc:535] server connection from 127.20.240.68:56203 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:43.905449 31139 connection.cc:535] client connection to 127.20.240.68:38371 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:43.905628 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 30727
W20240503 14:15:43.906157 30744 connection.cc:535] server connection from 127.20.240.68:51193 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:43.907385 31139 consensus_peers.cc:485] T 1af2fb3fb3fd4dfe873d124232d812b5 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.907524 31139 leader_election.cc:334] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111)
I20240503 14:15:43.907646 31139 leader_election.cc:302] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.907910 14390 raft_consensus.cc:2726] T 859456c340dc47bf8fc600ebe8c4b984 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.907989 31139 consensus_peers.cc:485] T bf316cde5631415faecbc1bb26690b79 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.908097 31139 leader_election.cc:334] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111)
I20240503 14:15:43.908192 31139 leader_election.cc:302] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.908408 14390 raft_consensus.cc:2726] T ff8d8c22d99c462b931a77297c5ff1ff P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.908537 31139 consensus_peers.cc:485] T 3fb8c05985f34deebc145c3c51dc413d P 530893a961af4b95b6b687fa5ed22f9d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.908682 31139 consensus_peers.cc:485] T 3a82a00c33e1412f93ab071d5b0d824a P 530893a961af4b95b6b687fa5ed22f9d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.908810 31139 consensus_peers.cc:485] T 6225866853d94dda88c87c6af053c739 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.908923 31139 consensus_peers.cc:485] T 1b4544aefcc54a828dc3b75726367f70 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.909039 31139 leader_election.cc:334] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111)
I20240503 14:15:43.909161 31139 leader_election.cc:302] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 5388902973134958a030fd63ab4faba4, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:43.909411 14390 raft_consensus.cc:2726] T 25eb1d6c445940d6bd8753a88381bb8c P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.909897 30746 consensus_peers.cc:485] T 609db8c3f3b245eaa31c82915b61188f P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.910082 30746 consensus_peers.cc:485] T f6cbcf7c01bb4265a7c8fd4c897c4928 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:43.916044 30850 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:15:43.920949 31008 consensus_peers.cc:485] T 81973d2082ab4bacb61722e51c146f30 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:43.932596 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:43.933485 31008 consensus_peers.cc:485] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.933976 31138 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:43.955757 31008 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:44.083547 31007 consensus_peers.cc:485] T a88d8b316d064c4e8fdccaba77d656c2 P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:44.108804 31007 consensus_peers.cc:485] T a3888675106d4d389718380627c95edf P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:44.148077 31008 consensus_peers.cc:485] T f7242c3728b84084a738fde7ae32ec0a P fde134d0fa2b4900b9badd2e08fea09d -> Peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Couldn't send request to peer 5388902973134958a030fd63ab4faba4. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.192718 31138 consensus_peers.cc:485] T d2bb66488d714d39900331f4a257e8b3 P 530893a961af4b95b6b687fa5ed22f9d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:44.264820 31203 raft_consensus.cc:1261] T 8dd90464f4b84baabd90a44a122e28b1 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Refusing update from remote peer d0dbfb0e43f948c3922926aacc365fc1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240503 14:15:44.265154 30745 consensus_peers.cc:462] T bca56ccf82a74df7bd8006e31e02bd20 P d0dbfb0e43f948c3922926aacc365fc1 -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
I20240503 14:15:44.266655 14389 consensus_queue.cc:1035] T 8dd90464f4b84baabd90a44a122e28b1 P d0dbfb0e43f948c3922926aacc365fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:15:44.272126 14358 raft_consensus.cc:482] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6409961ee8c0445687c11724213c44a9)
I20240503 14:15:44.272285 14358 raft_consensus.cc:504] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:44.272933 14358 leader_election.cc:288] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:44.276607 30745 leader_election.cc:334] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:44.277062 30744 leader_election.cc:334] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111)
I20240503 14:15:44.277194 30744 leader_election.cc:302] T f7bdba20d1844eefb6d3b29c5deb2873 P d0dbfb0e43f948c3922926aacc365fc1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0dbfb0e43f948c3922926aacc365fc1; no voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:44.277253 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 30727
I20240503 14:15:44.312139 14390 raft_consensus.cc:482] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6409961ee8c0445687c11724213c44a9)
I20240503 14:15:44.312366 14390 raft_consensus.cc:504] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:44.313186 14390 leader_election.cc:288] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:44.314517 31137 leader_election.cc:334] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111)
W20240503 14:15:44.314749 31139 leader_election.cc:334] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:44.314879 31139 leader_election.cc:302] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:44.315143 14390 raft_consensus.cc:2726] T 618ba33946424065a465217bae7cdbb0 P 530893a961af4b95b6b687fa5ed22f9d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:15:44.321509 14390 raft_consensus.cc:482] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6409961ee8c0445687c11724213c44a9)
I20240503 14:15:44.321677 14390 raft_consensus.cc:504] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "d0dbfb0e43f948c3922926aacc365fc1" member_type: VOTER last_known_addr { host: "127.20.240.69" port: 39689 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } }
I20240503 14:15:44.322633 14390 leader_election.cc:288] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:44.323508 31138 leader_election.cc:334] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:44.323930 31137 leader_election.cc:334] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111)
I20240503 14:15:44.324067 31137 leader_election.cc:302] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6409961ee8c0445687c11724213c44a9, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:44.324326 14390 raft_consensus.cc:2726] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:44.328948 31136 connection.cc:535] client connection to 127.20.240.69:39689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:44.329401 31005 connection.cc:535] client connection to 127.20.240.69:39689 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:44.329794 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 30989
W20240503 14:15:44.330332 31136 consensus_peers.cc:485] T 133f5691d69b461f8e7502ecffdb8a89 P 530893a961af4b95b6b687fa5ed22f9d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330401 31005 consensus_peers.cc:485] T 52683e63fe294a15aa301171025afe0a P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330448 31136 consensus_peers.cc:485] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330533 31005 consensus_peers.cc:485] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330539 31136 leader_election.cc:334] T a82470f287274b1c878089fa9d56ef40 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111)
W20240503 14:15:44.330639 31005 consensus_peers.cc:485] T e57c283674c342169940a5903516ec93 P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330754 31005 consensus_peers.cc:485] T f3dded25198e4d7d965fd9085e4ccd6d P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330776 31136 consensus_peers.cc:485] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330862 31005 consensus_peers.cc:485] T 4b6b6e0cac9a4ea6a0189970d88f9d72 P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.330983 31005 consensus_peers.cc:485] T 54a4dbdf6bd64453a9c2c78dbf550448 P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:44.335773 31111 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:44.335955 14390 raft_consensus.cc:482] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6409961ee8c0445687c11724213c44a9)
I20240503 14:15:44.336119 14390 raft_consensus.cc:504] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:44.336990 14390 leader_election.cc:288] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:44.338440 31137 leader_election.cc:334] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039): Network error: Client connection negotiation failed: client connection to 127.20.240.66:37039: connect: Connection refused (error 111)
W20240503 14:15:44.338665 31139 leader_election.cc:334] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
I20240503 14:15:44.338879 31139 leader_election.cc:302] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 530893a961af4b95b6b687fa5ed22f9d; no voters: 6409961ee8c0445687c11724213c44a9, 793a0d5dd1634c5faaf0e1e4851e8294
I20240503 14:15:44.339216 14390 raft_consensus.cc:2726] T 6ac53dcb19f941c48dd6c6cb623ccf58 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:44.507315 31139 consensus_peers.cc:462] T 7f807f1ac8d8417aaffd9254993b2085 P 530893a961af4b95b6b687fa5ed22f9d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
I20240503 14:15:44.664760 14409 raft_consensus.cc:482] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6409961ee8c0445687c11724213c44a9)
I20240503 14:15:44.664963 14409 raft_consensus.cc:504] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "793a0d5dd1634c5faaf0e1e4851e8294" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 33293 } }
I20240503 14:15:44.665678 14409 leader_election.cc:288] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293)
W20240503 14:15:44.669225 31005 consensus_peers.cc:485] T c9048a340c8c49f3be2c372aac871118 P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:44.670990 14431 raft_consensus.cc:482] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5388902973134958a030fd63ab4faba4)
I20240503 14:15:44.671167 14431 raft_consensus.cc:504] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:44.671789 14431 leader_election.cc:288] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371), a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889)
W20240503 14:15:44.673239 31005 consensus_peers.cc:485] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d -> Peer d0dbfb0e43f948c3922926aacc365fc1 (127.20.240.69:39689): Couldn't send request to peer d0dbfb0e43f948c3922926aacc365fc1. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.69:39689: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:15:44.673933 31007 consensus_peers.cc:485] T 0abfcd38ecc64acfac35c676b941581a P fde134d0fa2b4900b9badd2e08fea09d -> Peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Couldn't send request to peer 6bbc686598f24f518d7ca6bb2063058d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:15:44.675204 13825 raft_consensus.cc:482] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6409961ee8c0445687c11724213c44a9)
I20240503 14:15:44.675472 14409 raft_consensus.cc:482] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5388902973134958a030fd63ab4faba4)
I20240503 14:15:44.675622 14409 raft_consensus.cc:504] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } } peers { permanent_uuid: "5388902973134958a030fd63ab4faba4" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 38371 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } }
I20240503 14:15:44.676169 14409 leader_election.cc:288] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465), 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371)
I20240503 14:15:44.675415 13825 raft_consensus.cc:504] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a4cdf05256a0407c8929ac43eb55d44f" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 43889 } } peers { permanent_uuid: "fde134d0fa2b4900b9badd2e08fea09d" member_type: VOTER last_known_addr { host: "127.20.240.71" port: 34707 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } }
I20240503 14:15:44.676977 13825 leader_election.cc:288] T 92f507cdcfb04eb4a9eb448773206783 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889), 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039)
W20240503 14:15:44.679677 31008 leader_election.cc:334] T f11b07607b0e430db646172f863cdc1f P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 793a0d5dd1634c5faaf0e1e4851e8294 (127.20.240.65:33293): Network error: Client connection negotiation failed: client connection to 127.20.240.65:33293: connect: Connection refused (error 111)
W20240503 14:15:44.679900 31008 consensus_peers.cc:485] T 9202075be4bc472ebe781f577bb50ba9 P fde134d0fa2b4900b9badd2e08fea09d -> Peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Couldn't send request to peer a4cdf05256a0407c8929ac43eb55d44f. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:15:44.679908 31007 leader_election.cc:334] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465): Network error: Client connection negotiation failed: client connection to 127.20.240.70:39465: connect: Connection refused (error 111)
W20240503 14:15:44.680177 31008 leader_election.cc:334] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a4cdf05256a0407c8929ac43eb55d44f (127.20.240.67:43889): Network error: Client connection negotiation failed: client connection to 127.20.240.67:43889: connect: Connection refused (error 111)
W20240503 14:15:44.680456 31008 leader_election.cc:334] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111)
I20240503 14:15:44.680563 31008 leader_election.cc:302] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 5388902973134958a030fd63ab4faba4, a4cdf05256a0407c8929ac43eb55d44f
I20240503 14:15:44.680815 14428 raft_consensus.cc:2726] T d7ba7409f5bd4531bc5370019aca14c7 P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:15:44.681341 14428 raft_consensus.cc:482] T 48266bf4ad9b4976b639553490ffebbe P fde134d0fa2b4900b9badd2e08fea09d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5388902973134958a030fd63ab4faba4)
W20240503 14:15:44.681629 31008 leader_election.cc:334] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5388902973134958a030fd63ab4faba4 (127.20.240.68:38371): Network error: Client connection negotiation failed: client connection to 127.20.240.68:38371: connect: Connection refused (error 111)
I20240503 14:15:44.681735 31008 leader_election.cc:302] T ee914d743a2c46418f79728f3d32d1c5 P fde134d0fa2b4900b9badd2e08fea09d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fde134d0fa2b4900b9badd2e08fea09d; no voters: 5388902973134958a030fd63ab4faba4, 6bbc686598f24f518d7ca6bb2063058d
I20240503 14:15:44.681896 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 30989
W20240503 14:15:44.732947 31138 connection.cc:535] client connection to 127.20.240.71:34707 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:15:44.732950 31139 connection.cc:535] server connection from 127.20.240.71:53445 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:15:44.733620 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 31121
W20240503 14:15:44.733901 31138 consensus_peers.cc:485] T bde4e7b3ac8e47daba3dc5090e943aa7 P 530893a961af4b95b6b687fa5ed22f9d -> Peer fde134d0fa2b4900b9badd2e08fea09d (127.20.240.71:34707): Couldn't send request to peer fde134d0fa2b4900b9badd2e08fea09d. Status: Network error: Client connection negotiation failed: client connection to 127.20.240.71:34707: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:15:44.739735 31242 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:45.067973 14425 raft_consensus.cc:482] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6409961ee8c0445687c11724213c44a9)
I20240503 14:15:45.068189 14425 raft_consensus.cc:504] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "530893a961af4b95b6b687fa5ed22f9d" member_type: VOTER last_known_addr { host: "127.20.240.72" port: 41331 } } peers { permanent_uuid: "6409961ee8c0445687c11724213c44a9" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 37039 } } peers { permanent_uuid: "6bbc686598f24f518d7ca6bb2063058d" member_type: VOTER last_known_addr { host: "127.20.240.70" port: 39465 } }
I20240503 14:15:45.068878 14425 leader_election.cc:288] T 45da3dab7a204d378273a8985d91ace8 P 530893a961af4b95b6b687fa5ed22f9d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6409961ee8c0445687c11724213c44a9 (127.20.240.66:37039), 6bbc686598f24f518d7ca6bb2063058d (127.20.240.70:39465)
I20240503 14:15:45.069154 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 31121
I20240503 14:15:45.123028 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 27871
I20240503 14:15:45.128976 27932 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:45.436084 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 27871
2024-05-03T14:15:45Z chronyd exiting
I20240503 14:15:45.504285 21441 test_util.cc:175] -----------------------------------------------
I20240503 14:15:45.504377 21441 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714745515740426-21441-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (176317 ms)
[----------] 1 test from DDLDuringRebalancingTest (176317 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240503 14:15:45.506531 21441 test_util.cc:268] Using random seed: 1527414977
I20240503 14:15:45.508306 21441 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:15:45.508381 21441 ts_itest-base.cc:113] --------------
I20240503 14:15:45.508411 21441 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:15:45.508460 21441 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:15:45.508508 21441 ts_itest-base.cc:116] --------------
2024-05-03T14:15:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:15:45Z Disabled control of system clock
I20240503 14:15:45.537492 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:44017
--webserver_interface=127.20.240.126
--webserver_port=0
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.240.126:44017
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240503 14:15:45.857997 14453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:45.858331 14453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:45.858471 14453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:45.865774 14453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:45.865888 14453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:15:45.865973 14453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:45.866046 14453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:15:45.866117 14453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:15:45.874889 14453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.20.240.126:44017
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.240.126:44017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.240.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:45.876113 14453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:45.877753 14453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:45.888128 14458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:45.888671 14459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:45.889245 14461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:45.890101 14453 server_base.cc:1034] running on GCE node
I20240503 14:15:45.890764 14453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:45.891988 14453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:45.893182 14453 hybrid_clock.cc:648] HybridClock initialized: now 1714745745893149 us; error 26 us; skew 500 ppm
I20240503 14:15:45.893479 14453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:45.900696 14453 webserver.cc:460] Webserver started at http://127.20.240.126:46589/ using document root <none> and password file <none>
I20240503 14:15:45.901407 14453 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:45.901520 14453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:45.901809 14453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:45.904294 14453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ad9605477a754b499b492bc01d1a6dd3"
format_stamp: "Formatted at 2024-05-03 14:15:45 on dist-test-slave-xhmt"
I20240503 14:15:45.905058 14453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ad9605477a754b499b492bc01d1a6dd3"
format_stamp: "Formatted at 2024-05-03 14:15:45 on dist-test-slave-xhmt"
I20240503 14:15:45.910019 14453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240503 14:15:45.913256 14467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:45.914557 14453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:45.914710 14453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ad9605477a754b499b492bc01d1a6dd3"
format_stamp: "Formatted at 2024-05-03 14:15:45 on dist-test-slave-xhmt"
I20240503 14:15:45.914884 14453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:45.933897 14453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:45.934631 14453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:45.934890 14453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:45.952823 14453 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.126:44017
I20240503 14:15:45.952847 14518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.126:44017 every 8 connection(s)
I20240503 14:15:45.954182 14453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:15:45.955803 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 14453
I20240503 14:15:45.956044 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:15:45.958578 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:45.971043 14519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3: Bootstrap starting.
I20240503 14:15:45.973732 14519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:45.974776 14519 log.cc:824] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:45.976991 14519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3: No bootstrap required, opened a new log
I20240503 14:15:45.982618 14519 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9605477a754b499b492bc01d1a6dd3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44017 } }
I20240503 14:15:45.982923 14519 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:45.983039 14519 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9605477a754b499b492bc01d1a6dd3, State: Initialized, Role: FOLLOWER
I20240503 14:15:45.983589 14519 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9605477a754b499b492bc01d1a6dd3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44017 } }
I20240503 14:15:45.983780 14519 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:15:45.983903 14519 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:15:45.984045 14519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:45.986013 14519 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9605477a754b499b492bc01d1a6dd3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44017 } }
I20240503 14:15:45.986498 14519 leader_election.cc:302] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad9605477a754b499b492bc01d1a6dd3; no voters:
I20240503 14:15:45.987058 14519 leader_election.cc:288] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:15:45.987175 14524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:45.987805 14524 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [term 1 LEADER]: Becoming Leader. State: Replica: ad9605477a754b499b492bc01d1a6dd3, State: Running, Role: LEADER
I20240503 14:15:45.988324 14524 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9605477a754b499b492bc01d1a6dd3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44017 } }
I20240503 14:15:45.988992 14519 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:15:45.991333 14525 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad9605477a754b499b492bc01d1a6dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9605477a754b499b492bc01d1a6dd3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44017 } } }
I20240503 14:15:45.991894 14525 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:45.991961 14526 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad9605477a754b499b492bc01d1a6dd3. Latest consensus state: current_term: 1 leader_uuid: "ad9605477a754b499b492bc01d1a6dd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad9605477a754b499b492bc01d1a6dd3" member_type: VOTER last_known_addr { host: "127.20.240.126" port: 44017 } } }
I20240503 14:15:45.992368 14526 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3 [sys.catalog]: This master's current role is: LEADER
I20240503 14:15:45.999241 14534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:15:46.003926 14534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:15:46.011801 14534 catalog_manager.cc:1347] Generated new cluster ID: fcfb90fd8150418b8272a2bc99bc2e56
I20240503 14:15:46.011894 14534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:15:46.023550 14534 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:15:46.025079 14534 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:15:46.040675 14534 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ad9605477a754b499b492bc01d1a6dd3: Generated new TSK 0
I20240503 14:15:46.041345 14534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:15:46.049710 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.65:0
--local_ip_for_outbound_sockets=127.20.240.65
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:15:46.371955 14543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:46.372222 14543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:46.372347 14543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:46.379573 14543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:46.379676 14543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:46.379834 14543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.65
I20240503 14:15:46.388459 14543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.240.65
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.65
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:46.389739 14543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:46.391489 14543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:46.403160 14548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:46.403775 14549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:46.404315 14551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:46.405051 14543 server_base.cc:1034] running on GCE node
I20240503 14:15:46.405604 14543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:46.406526 14543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:46.407721 14543 hybrid_clock.cc:648] HybridClock initialized: now 1714745746407662 us; error 58 us; skew 500 ppm
I20240503 14:15:46.408010 14543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:46.410072 14543 webserver.cc:460] Webserver started at http://127.20.240.65:38629/ using document root <none> and password file <none>
I20240503 14:15:46.410830 14543 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:46.410938 14543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:46.411237 14543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:46.413666 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e9d4c39af431443189e8475561596810"
format_stamp: "Formatted at 2024-05-03 14:15:46 on dist-test-slave-xhmt"
I20240503 14:15:46.414424 14543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e9d4c39af431443189e8475561596810"
format_stamp: "Formatted at 2024-05-03 14:15:46 on dist-test-slave-xhmt"
I20240503 14:15:46.419075 14543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 14:15:46.422196 14557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:46.423453 14543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:15:46.423597 14543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e9d4c39af431443189e8475561596810"
format_stamp: "Formatted at 2024-05-03 14:15:46 on dist-test-slave-xhmt"
I20240503 14:15:46.423774 14543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:46.452369 14543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:46.453104 14543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:46.453336 14543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:46.454150 14543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:46.455516 14543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:46.455611 14543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:46.455722 14543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:46.455796 14543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:46.485603 14543 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.65:32989
I20240503 14:15:46.485666 14669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.65:32989 every 8 connection(s)
I20240503 14:15:46.486861 14543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:15:46.492138 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 14543
I20240503 14:15:46.492318 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:15:46.495669 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.66:0
--local_ip_for_outbound_sockets=127.20.240.66
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:46.499600 14670 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:46.499873 14670 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:46.500571 14670 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:46.502614 14484 ts_manager.cc:194] Registered new tserver with Master: e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:46.504791 14484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.65:56631
W20240503 14:15:46.813714 14674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:46.813970 14674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:46.814062 14674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:46.821115 14674 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:46.821219 14674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:46.821343 14674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.66
I20240503 14:15:46.829962 14674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.240.66
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.66
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:46.831015 14674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:46.832580 14674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:46.843700 14682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:46.844110 14679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:46.844738 14680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:46.845783 14674 server_base.cc:1034] running on GCE node
I20240503 14:15:46.846305 14674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:46.847070 14674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:46.848269 14674 hybrid_clock.cc:648] HybridClock initialized: now 1714745746848221 us; error 25 us; skew 500 ppm
I20240503 14:15:46.848551 14674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:46.850600 14674 webserver.cc:460] Webserver started at http://127.20.240.66:45451/ using document root <none> and password file <none>
I20240503 14:15:46.851258 14674 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:46.851358 14674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:46.851642 14674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:46.853971 14674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
format_stamp: "Formatted at 2024-05-03 14:15:46 on dist-test-slave-xhmt"
I20240503 14:15:46.854683 14674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
format_stamp: "Formatted at 2024-05-03 14:15:46 on dist-test-slave-xhmt"
I20240503 14:15:46.859200 14674 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 14:15:46.862128 14688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:46.863382 14674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240503 14:15:46.863499 14674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
format_stamp: "Formatted at 2024-05-03 14:15:46 on dist-test-slave-xhmt"
I20240503 14:15:46.863649 14674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:46.893810 14674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:46.894559 14674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:46.894805 14674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:46.895665 14674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:46.897061 14674 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:46.897158 14674 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:46.897241 14674 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:46.897298 14674 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:46.927503 14674 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.66:36573
I20240503 14:15:46.927560 14800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.66:36573 every 8 connection(s)
I20240503 14:15:46.928654 14674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:15:46.936988 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 14674
I20240503 14:15:46.937214 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:15:46.940713 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.67:0
--local_ip_for_outbound_sockets=127.20.240.67
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:46.940800 14801 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:46.941061 14801 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:46.941705 14801 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:46.942982 14484 ts_manager.cc:194] Registered new tserver with Master: c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:46.943977 14484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.66:48811
W20240503 14:15:47.259276 14804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:47.259538 14804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:47.259672 14804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:47.266701 14804 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:47.266799 14804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:47.266971 14804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.67
I20240503 14:15:47.276177 14804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.240.67
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.67
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:47.277401 14804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:47.279084 14804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:47.290910 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:47.291359 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:47.292069 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:47.292830 14804 server_base.cc:1034] running on GCE node
I20240503 14:15:47.293349 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:47.294173 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:47.295384 14804 hybrid_clock.cc:648] HybridClock initialized: now 1714745747295342 us; error 38 us; skew 500 ppm
I20240503 14:15:47.295687 14804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:47.297762 14804 webserver.cc:460] Webserver started at http://127.20.240.67:46233/ using document root <none> and password file <none>
I20240503 14:15:47.298491 14804 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:47.298612 14804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:47.298910 14804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:47.301299 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b09f5cddfdf44e989f47a06c8da867f8"
format_stamp: "Formatted at 2024-05-03 14:15:47 on dist-test-slave-xhmt"
I20240503 14:15:47.302026 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b09f5cddfdf44e989f47a06c8da867f8"
format_stamp: "Formatted at 2024-05-03 14:15:47 on dist-test-slave-xhmt"
I20240503 14:15:47.306727 14804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 14:15:47.309865 14818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:47.311093 14804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:47.311235 14804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b09f5cddfdf44e989f47a06c8da867f8"
format_stamp: "Formatted at 2024-05-03 14:15:47 on dist-test-slave-xhmt"
I20240503 14:15:47.311407 14804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:47.328495 14804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:47.329170 14804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:47.329425 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:47.330273 14804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:47.331570 14804 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:47.331655 14804 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:47.331760 14804 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:47.331832 14804 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:47.360857 14804 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.67:44349
I20240503 14:15:47.360920 14930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.67:44349 every 8 connection(s)
I20240503 14:15:47.362109 14804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:15:47.367120 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 14804
I20240503 14:15:47.367286 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:15:47.370110 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.68:0
--local_ip_for_outbound_sockets=127.20.240.68
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:47.374102 14931 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:47.374392 14931 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:47.375070 14931 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:47.376331 14484 ts_manager.cc:194] Registered new tserver with Master: b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:15:47.377106 14484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.67:59147
I20240503 14:15:47.507560 14670 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
W20240503 14:15:47.687976 14934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:47.688237 14934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:47.688366 14934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:47.695629 14934 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:47.695734 14934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:47.695891 14934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.68
I20240503 14:15:47.704548 14934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.240.68
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.68
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:47.705693 14934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:47.707474 14934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:47.718955 14939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:47.719514 14942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:47.720345 14940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:47.721174 14934 server_base.cc:1034] running on GCE node
I20240503 14:15:47.721647 14934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:47.722465 14934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:47.723655 14934 hybrid_clock.cc:648] HybridClock initialized: now 1714745747723615 us; error 39 us; skew 500 ppm
I20240503 14:15:47.723925 14934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:47.726012 14934 webserver.cc:460] Webserver started at http://127.20.240.68:45147/ using document root <none> and password file <none>
I20240503 14:15:47.726716 14934 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:47.726827 14934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:47.727124 14934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:47.729575 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "49e64758760942e1abdcdc0d2e1f07dd"
format_stamp: "Formatted at 2024-05-03 14:15:47 on dist-test-slave-xhmt"
I20240503 14:15:47.730337 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "49e64758760942e1abdcdc0d2e1f07dd"
format_stamp: "Formatted at 2024-05-03 14:15:47 on dist-test-slave-xhmt"
I20240503 14:15:47.734755 14934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:15:47.737704 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:47.738945 14934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:15:47.739094 14934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "49e64758760942e1abdcdc0d2e1f07dd"
format_stamp: "Formatted at 2024-05-03 14:15:47 on dist-test-slave-xhmt"
I20240503 14:15:47.739284 14934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:47.756803 14934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:47.757509 14934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:47.757767 14934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:47.758626 14934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:47.759972 14934 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:47.760076 14934 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:47.760188 14934 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:47.760267 14934 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:47.791663 14934 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.68:36453
I20240503 14:15:47.791740 15060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.68:36453 every 8 connection(s)
I20240503 14:15:47.792927 14934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:15:47.801054 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 14934
I20240503 14:15:47.801221 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:15:47.804803 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:0
--local_ip_for_outbound_sockets=127.20.240.69
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:47.805183 15061 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:47.805500 15061 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:47.806399 15061 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:47.807798 14484 ts_manager.cc:194] Registered new tserver with Master: 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:47.808728 14484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.68:54145
I20240503 14:15:47.946126 14801 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
W20240503 14:15:48.122118 15064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:48.122431 15064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:48.122573 15064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:48.129909 15064 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:48.130026 15064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:48.130170 15064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:15:48.138787 15064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:48.140041 15064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:48.141666 15064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:48.153157 15070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:48.153788 15069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:48.154505 15072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:48.155192 15064 server_base.cc:1034] running on GCE node
I20240503 14:15:48.155745 15064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:48.156567 15064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:48.157763 15064 hybrid_clock.cc:648] HybridClock initialized: now 1714745748157713 us; error 40 us; skew 500 ppm
I20240503 14:15:48.158052 15064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:48.160147 15064 webserver.cc:460] Webserver started at http://127.20.240.69:43137/ using document root <none> and password file <none>
I20240503 14:15:48.160799 15064 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:48.160900 15064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:48.161212 15064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:48.163601 15064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e1e5858c55de4939907c3253a25fad0d"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:15:48.164301 15064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e1e5858c55de4939907c3253a25fad0d"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:15:48.168990 15064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240503 14:15:48.171986 15078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:48.173182 15064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:48.173312 15064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e1e5858c55de4939907c3253a25fad0d"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:15:48.173520 15064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:48.196507 15064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:48.197300 15064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:48.197592 15064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:48.198567 15064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:48.200093 15064 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:48.200188 15064 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:48.200322 15064 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:48.200416 15064 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:48.231604 15064 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:35389
I20240503 14:15:48.231668 15190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:35389 every 8 connection(s)
I20240503 14:15:48.232843 15064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:15:48.236793 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 15064
I20240503 14:15:48.236953 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:15:48.240145 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.70:0
--local_ip_for_outbound_sockets=127.20.240.70
--webserver_interface=127.20.240.70
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:48.245507 15191 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:48.245774 15191 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:48.246595 15191 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:48.248061 14484 ts_manager.cc:194] Registered new tserver with Master: e1e5858c55de4939907c3253a25fad0d (127.20.240.69:35389)
I20240503 14:15:48.249150 14484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:48251
I20240503 14:15:48.379356 14931 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
W20240503 14:15:48.567490 15194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:48.567786 15194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:48.567931 15194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:48.575156 15194 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:48.575286 15194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:48.575445 15194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.70
I20240503 14:15:48.584005 15194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.240.70
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.70
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:48.585209 15194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:48.586895 15194 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:48.598600 15200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:48.599167 15199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:48.599990 15202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:48.600868 15194 server_base.cc:1034] running on GCE node
I20240503 14:15:48.601444 15194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:48.602375 15194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:48.603596 15194 hybrid_clock.cc:648] HybridClock initialized: now 1714745748603560 us; error 34 us; skew 500 ppm
I20240503 14:15:48.603907 15194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:48.606367 15194 webserver.cc:460] Webserver started at http://127.20.240.70:41769/ using document root <none> and password file <none>
I20240503 14:15:48.607136 15194 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:48.607250 15194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:48.607686 15194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:48.610282 15194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "19297c5910bf4b03b12bc4d7168475e2"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:15:48.611053 15194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "19297c5910bf4b03b12bc4d7168475e2"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:15:48.616022 15194 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:15:48.619441 15208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:48.620666 15194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:15:48.620813 15194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "19297c5910bf4b03b12bc4d7168475e2"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:15:48.620999 15194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:48.650158 15194 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:48.650928 15194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:48.651173 15194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:48.652033 15194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:48.653384 15194 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:48.653481 15194 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:48.653590 15194 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:48.653666 15194 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:48.684093 15194 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.70:45231
I20240503 14:15:48.684162 15320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.70:45231 every 8 connection(s)
I20240503 14:15:48.685312 15194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:15:48.692576 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 15194
I20240503 14:15:48.692756 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:15:48.695925 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.71:0
--local_ip_for_outbound_sockets=127.20.240.71
--webserver_interface=127.20.240.71
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:48.697552 15321 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:48.697824 15321 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:48.698575 15321 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:48.699944 14484 ts_manager.cc:194] Registered new tserver with Master: 19297c5910bf4b03b12bc4d7168475e2 (127.20.240.70:45231)
I20240503 14:15:48.701027 14484 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.70:44351
I20240503 14:15:48.813613 15061 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
W20240503 14:15:49.019846 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:49.020274 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:49.020432 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:49.027925 15324 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:49.028062 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:49.028224 15324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.71
I20240503 14:15:49.037014 15324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.240.71
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.71
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:49.038276 15324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:49.039990 15324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:49.051813 15329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:49.052618 15330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:49.053009 15332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:49.053820 15324 server_base.cc:1034] running on GCE node
I20240503 14:15:49.054431 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:49.055306 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:49.056524 15324 hybrid_clock.cc:648] HybridClock initialized: now 1714745749056464 us; error 40 us; skew 500 ppm
I20240503 14:15:49.056805 15324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:49.059074 15324 webserver.cc:460] Webserver started at http://127.20.240.71:34901/ using document root <none> and password file <none>
I20240503 14:15:49.059720 15324 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:49.059835 15324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:49.060125 15324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:49.062490 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "d8ecf8e658bc4102b272f90b8146cb99"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:15:49.063202 15324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "d8ecf8e658bc4102b272f90b8146cb99"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:15:49.068588 15324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:15:49.071498 15338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.072585 15324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:49.072738 15324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d8ecf8e658bc4102b272f90b8146cb99"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:15:49.072912 15324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:49.089761 15324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:49.090492 15324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:49.090725 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:49.091651 15324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:49.093166 15324 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:49.093261 15324 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.093365 15324 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:49.093444 15324 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.124100 15324 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.71:41887
I20240503 14:15:49.124172 15450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.71:41887 every 8 connection(s)
I20240503 14:15:49.125361 15324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:15:49.134521 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 15324
I20240503 14:15:49.134717 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:15:49.137319 15451 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:49.137691 15451 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:49.138119 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.72:0
--local_ip_for_outbound_sockets=127.20.240.72
--webserver_interface=127.20.240.72
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:15:49.138592 15451 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:49.140050 14483 ts_manager.cc:194] Registered new tserver with Master: d8ecf8e658bc4102b272f90b8146cb99 (127.20.240.71:41887)
I20240503 14:15:49.141237 14483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.71:37449
I20240503 14:15:49.251955 15191 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
W20240503 14:15:49.462085 15454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:49.462401 15454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:15:49.462539 15454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:49.469775 15454 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:15:49.469903 15454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:15:49.470063 15454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.72
I20240503 14:15:49.479151 15454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.240.72
--webserver_port=0
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.72
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:15:49.480334 15454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:15:49.481904 15454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:15:49.493992 15459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:49.494747 15460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:15:49.495214 15462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:15:49.496333 15454 server_base.cc:1034] running on GCE node
I20240503 14:15:49.496855 15454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:15:49.497653 15454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:15:49.498845 15454 hybrid_clock.cc:648] HybridClock initialized: now 1714745749498793 us; error 45 us; skew 500 ppm
I20240503 14:15:49.499122 15454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:15:49.501161 15454 webserver.cc:460] Webserver started at http://127.20.240.72:36977/ using document root <none> and password file <none>
I20240503 14:15:49.501799 15454 fs_manager.cc:359] Metadata directory not provided
I20240503 14:15:49.501891 15454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:15:49.502159 15454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:15:49.504549 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ebe953d95ec24ec7a9071fb834984cce"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:15:49.505295 15454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ebe953d95ec24ec7a9071fb834984cce"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:15:49.510149 15454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240503 14:15:49.513470 15468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.514714 15454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:15:49.514838 15454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ebe953d95ec24ec7a9071fb834984cce"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:15:49.515000 15454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:15:49.540926 15454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:15:49.541643 15454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:15:49.541877 15454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:15:49.542804 15454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:15:49.544257 15454 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:15:49.544359 15454 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.544446 15454 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:15:49.544500 15454 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:15:49.574472 15454 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.72:43695
I20240503 14:15:49.574534 15580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.72:43695 every 8 connection(s)
I20240503 14:15:49.575724 15454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:15:49.579113 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 15454
I20240503 14:15:49.579286 21441 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:15:49.587497 15581 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:15:49.587743 15581 heartbeater.cc:464] Registering TS with master...
I20240503 14:15:49.588380 15581 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:15:49.589569 14483 ts_manager.cc:194] Registered new tserver with Master: ebe953d95ec24ec7a9071fb834984cce (127.20.240.72:43695)
I20240503 14:15:49.590406 14483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.72:34657
I20240503 14:15:49.594630 21441 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:15:49.610422 21441 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:15:49.610529 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 15064
I20240503 14:15:49.618132 15186 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:49.703387 15321 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
I20240503 14:15:49.704988 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 15064
I20240503 14:15:49.730059 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 15194
I20240503 14:15:49.739969 15316 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:49.820946 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 15194
I20240503 14:15:49.838070 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 15324
I20240503 14:15:49.847404 15446 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:49.926709 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 15324
I20240503 14:15:49.944805 21441 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu pid 15454
I20240503 14:15:49.953169 15576 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:15:50.038058 21441 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu with pid 15454
I20240503 14:15:53.806602 14483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:53.844375 14866 tablet_service.cc:1467] Processing CreateTablet for tablet dc909a5952a944bfbc49a8052f73359c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.845615 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 67cbe83d27cc4107b88da1f87c5468e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.846765 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67cbe83d27cc4107b88da1f87c5468e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.849870 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc909a5952a944bfbc49a8052f73359c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.849956 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 3aab5902c7cd4daaada285f05615dfad (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.852092 14996 tablet_service.cc:1467] Processing CreateTablet for tablet 3aab5902c7cd4daaada285f05615dfad (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.853192 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aab5902c7cd4daaada285f05615dfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.854064 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aab5902c7cd4daaada285f05615dfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.852275 14736 tablet_service.cc:1467] Processing CreateTablet for tablet dc909a5952a944bfbc49a8052f73359c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.853132 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 67cbe83d27cc4107b88da1f87c5468e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.854658 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc909a5952a944bfbc49a8052f73359c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.857335 14605 tablet_service.cc:1467] Processing CreateTablet for tablet dc909a5952a944bfbc49a8052f73359c (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.858033 14604 tablet_service.cc:1467] Processing CreateTablet for tablet 3aab5902c7cd4daaada285f05615dfad (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.858722 14604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aab5902c7cd4daaada285f05615dfad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.858899 14603 tablet_service.cc:1467] Processing CreateTablet for tablet 67cbe83d27cc4107b88da1f87c5468e2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=c1b96937ddd64f40a0de09bd2c0e02d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:53.859339 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67cbe83d27cc4107b88da1f87c5468e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.860527 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc909a5952a944bfbc49a8052f73359c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.866396 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67cbe83d27cc4107b88da1f87c5468e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:53.885924 15611 tablet_bootstrap.cc:492] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:53.886960 15610 tablet_bootstrap.cc:492] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:53.891604 15610 tablet_bootstrap.cc:654] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.893369 15610 log.cc:824] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:53.894418 15612 tablet_bootstrap.cc:492] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:53.895676 15611 tablet_bootstrap.cc:654] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.896209 15609 tablet_bootstrap.cc:492] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:53.896852 15611 log.cc:824] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:53.897125 15610 tablet_bootstrap.cc:492] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:53.897451 15610 ts_tablet_manager.cc:1367] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240503 14:15:53.897740 15612 tablet_bootstrap.cc:654] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.899196 15612 log.cc:824] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:53.900043 15609 tablet_bootstrap.cc:654] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.901475 15609 log.cc:824] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8: Log is configured to *not* fsync() on all Append() calls
I20240503 14:15:53.909324 15612 tablet_bootstrap.cc:492] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:53.909368 15609 tablet_bootstrap.cc:492] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:53.909368 15611 tablet_bootstrap.cc:492] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:53.909655 15609 ts_tablet_manager.cc:1367] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240503 14:15:53.909672 15612 ts_tablet_manager.cc:1367] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20240503 14:15:53.909691 15611 ts_tablet_manager.cc:1367] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.008s
I20240503 14:15:53.909849 15610 raft_consensus.cc:348] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.910341 15610 raft_consensus.cc:374] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.910490 15610 raft_consensus.cc:729] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.911234 15610 consensus_queue.cc:260] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.918807 15609 raft_consensus.cc:348] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.918807 15611 raft_consensus.cc:348] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:53.919250 15611 raft_consensus.cc:374] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.919286 15609 raft_consensus.cc:374] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.919466 15609 raft_consensus.cc:729] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.919457 15611 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.920189 15611 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:53.920189 15609 consensus_queue.cc:260] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.922437 15610 ts_tablet_manager.cc:1398] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.025s user 0.008s sys 0.015s
I20240503 14:15:53.922982 15610 tablet_bootstrap.cc:492] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:53.922931 15612 raft_consensus.cc:348] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.923516 15612 raft_consensus.cc:374] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.923681 15612 raft_consensus.cc:729] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.924468 15612 consensus_queue.cc:260] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.924729 15610 tablet_bootstrap.cc:654] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.933616 15611 ts_tablet_manager.cc:1398] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.024s user 0.011s sys 0.003s
I20240503 14:15:53.934072 15611 tablet_bootstrap.cc:492] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810: Bootstrap starting.
W20240503 14:15:53.935037 14802 tablet.cc:2369] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:53.935895 15611 tablet_bootstrap.cc:654] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.939484 15609 ts_tablet_manager.cc:1398] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.030s user 0.010s sys 0.006s
I20240503 14:15:53.941022 15612 ts_tablet_manager.cc:1398] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.031s user 0.009s sys 0.010s
I20240503 14:15:53.941474 15612 tablet_bootstrap.cc:492] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:53.944161 15612 tablet_bootstrap.cc:654] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.944505 15609 tablet_bootstrap.cc:492] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:53.945200 15611 tablet_bootstrap.cc:492] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:53.945421 15611 ts_tablet_manager.cc:1367] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 14:15:53.946270 15609 tablet_bootstrap.cc:654] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.946503 15611 raft_consensus.cc:348] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.946812 15611 raft_consensus.cc:374] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.946957 15611 raft_consensus.cc:729] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.947332 15611 consensus_queue.cc:260] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.948691 15611 ts_tablet_manager.cc:1398] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:53.948738 15610 tablet_bootstrap.cc:492] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:53.948881 15610 ts_tablet_manager.cc:1367] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.009s
I20240503 14:15:53.949071 15611 tablet_bootstrap.cc:492] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:53.949927 15610 raft_consensus.cc:348] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:53.950086 15610 raft_consensus.cc:374] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.950219 15610 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.950603 15610 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:53.951117 15612 tablet_bootstrap.cc:492] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:53.951181 15610 ts_tablet_manager.cc:1398] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:53.951303 15612 ts_tablet_manager.cc:1367] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240503 14:15:53.952105 15612 raft_consensus.cc:348] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.952262 15612 raft_consensus.cc:374] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.952384 15612 raft_consensus.cc:729] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.952636 15612 consensus_queue.cc:260] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.953151 15612 ts_tablet_manager.cc:1398] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:53.954030 15611 tablet_bootstrap.cc:654] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:53.956614 15611 tablet_bootstrap.cc:492] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:53.956770 15611 ts_tablet_manager.cc:1367] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:15:53.957053 15609 tablet_bootstrap.cc:492] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:53.957201 15609 ts_tablet_manager.cc:1367] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240503 14:15:53.957474 15611 raft_consensus.cc:348] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.957688 15611 raft_consensus.cc:374] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.957772 15611 raft_consensus.cc:729] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.958010 15611 consensus_queue.cc:260] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:53.958168 15609 raft_consensus.cc:348] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:53.958410 15609 raft_consensus.cc:374] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:53.958532 15609 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:53.958608 15611 ts_tablet_manager.cc:1398] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:53.958777 15609 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:53.959300 15609 ts_tablet_manager.cc:1398] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240503 14:15:53.993506 14671 tablet.cc:2369] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:54.007573 15619 raft_consensus.cc:482] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:54.007851 15619 raft_consensus.cc:504] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:54.009538 15619 leader_election.cc:288] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:54.017058 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:54.017110 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:15:54.017535 14886 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:54.017566 14756 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:54.018100 14559 leader_election.cc:302] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:54.018484 15619 raft_consensus.cc:2781] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:54.018630 15619 raft_consensus.cc:482] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:54.018738 15619 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.020937 15619 raft_consensus.cc:504] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:54.021497 15619 leader_election.cc:288] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:54.021850 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:54.021973 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:15:54.022029 14886 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.022157 14756 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.024080 14886 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:15:54.024108 14756 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:15:54.024464 14559 leader_election.cc:302] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:54.024768 15619 raft_consensus.cc:2781] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:54.025172 15619 raft_consensus.cc:686] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:15:54.025676 15619 consensus_queue.cc:237] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:54.029515 14483 catalog_manager.cc:5484] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
W20240503 14:15:54.048426 15062 tablet.cc:2369] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:15:54.117976 14932 tablet.cc:2369] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:15:54.118896 15619 raft_consensus.cc:482] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:54.119108 15619 raft_consensus.cc:504] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:54.119820 15619 leader_election.cc:288] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:54.120410 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:15:54.120680 14756 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:54.121126 14559 leader_election.cc:302] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:54.121454 15619 raft_consensus.cc:2781] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:54.121570 15619 raft_consensus.cc:482] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:54.121682 15619 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.124037 15619 raft_consensus.cc:504] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:54.124651 15619 leader_election.cc:288] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:54.125075 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:15:54.125284 14756 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.126273 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:15:54.126341 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:15:54.126796 15016 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:54.127954 14756 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:15:54.128276 14559 leader_election.cc:302] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:15:54.128535 15619 raft_consensus.cc:2781] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:54.128697 15619 raft_consensus.cc:686] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:15:54.128945 15619 consensus_queue.cc:237] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:54.131325 14483 catalog_manager.cc:5484] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:15:54.132954 15619 raft_consensus.cc:482] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:54.133080 15619 raft_consensus.cc:504] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:54.134073 15619 leader_election.cc:288] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:54.134110 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:54.134398 14886 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:54.134436 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:15:54.134654 15015 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:54.134790 14558 leader_election.cc:302] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:54.135047 15619 raft_consensus.cc:2781] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:54.135140 15619 raft_consensus.cc:482] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:54.135255 15619 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.137285 15619 raft_consensus.cc:504] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:54.137770 15619 leader_election.cc:288] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:54.138120 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:54.138180 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:15:54.138338 14886 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.138381 15015 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.140102 14886 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:15:54.140367 15015 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:15:54.140461 14558 leader_election.cc:302] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:54.140734 15619 raft_consensus.cc:2781] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:54.140862 15619 raft_consensus.cc:686] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:15:54.141104 15619 consensus_queue.cc:237] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:54.143242 14483 catalog_manager.cc:5484] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:15:54.456691 15619 consensus_queue.cc:1035] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:54.468247 15628 consensus_queue.cc:1035] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:54.537225 15632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:54.537510 15632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:54.547587 15632 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:54.563058 15628 consensus_queue.cc:1035] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:54.569734 15619 consensus_queue.cc:1035] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:54.583390 15619 consensus_queue.cc:1035] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 14:15:54.587278 15015 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:54.590488 15619 consensus_queue.cc:1035] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 222.551 ms
time per row: 0.111276 ms
I20240503 14:15:54.939579 14481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:54.948557 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 72b8519d872146b98f942fe73c5e5e84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.948927 14736 tablet_service.cc:1467] Processing CreateTablet for tablet dd6bbdd8c1d84a6d88e4810b0f125cf5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.949045 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b8519d872146b98f942fe73c5e5e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.949064 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 72b8519d872146b98f942fe73c5e5e84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.949224 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 72b8519d872146b98f942fe73c5e5e84 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.949348 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6bbdd8c1d84a6d88e4810b0f125cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.949440 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b8519d872146b98f942fe73c5e5e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.949581 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72b8519d872146b98f942fe73c5e5e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.950769 14866 tablet_service.cc:1467] Processing CreateTablet for tablet dd6bbdd8c1d84a6d88e4810b0f125cf5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.951153 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6bbdd8c1d84a6d88e4810b0f125cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.951180 14734 tablet_service.cc:1467] Processing CreateTablet for tablet c8c2fc935476474182cf1bf00c15b547 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.951603 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c2fc935476474182cf1bf00c15b547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.956807 14605 tablet_service.cc:1467] Processing CreateTablet for tablet c8c2fc935476474182cf1bf00c15b547 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.957162 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c2fc935476474182cf1bf00c15b547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.962863 14996 tablet_service.cc:1467] Processing CreateTablet for tablet dd6bbdd8c1d84a6d88e4810b0f125cf5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.963248 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd6bbdd8c1d84a6d88e4810b0f125cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.963178 14864 tablet_service.cc:1467] Processing CreateTablet for tablet c8c2fc935476474182cf1bf00c15b547 (DEFAULT_TABLE table=rebalance_test_table_1 [id=264a4ecc21ca4a2898a436aa933c8cad]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:54.963609 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8c2fc935476474182cf1bf00c15b547. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:54.965449 15691 tablet_bootstrap.cc:492] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:54.967314 15690 tablet_bootstrap.cc:492] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:54.968246 15691 tablet_bootstrap.cc:654] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:54.969622 15690 tablet_bootstrap.cc:654] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:54.975294 15695 tablet_bootstrap.cc:492] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:54.978179 15690 tablet_bootstrap.cc:492] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:54.978477 15690 ts_tablet_manager.cc:1367] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.006s
I20240503 14:15:54.979158 15694 tablet_bootstrap.cc:492] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:54.979804 15690 raft_consensus.cc:348] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:54.980206 15690 raft_consensus.cc:374] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:54.980367 15690 raft_consensus.cc:729] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:54.980751 15690 consensus_queue.cc:260] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:54.982002 15690 ts_tablet_manager.cc:1398] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:54.982206 15694 tablet_bootstrap.cc:654] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:54.982334 15691 tablet_bootstrap.cc:492] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:54.982528 15691 ts_tablet_manager.cc:1367] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.005s
I20240503 14:15:54.983387 15691 raft_consensus.cc:348] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:54.983660 15691 raft_consensus.cc:374] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:54.983805 15691 raft_consensus.cc:729] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:54.983952 15695 tablet_bootstrap.cc:654] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:54.984118 15691 consensus_queue.cc:260] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:54.984336 15690 tablet_bootstrap.cc:492] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:54.985925 15690 tablet_bootstrap.cc:654] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:54.986963 15691 ts_tablet_manager.cc:1398] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:15:54.987308 15691 tablet_bootstrap.cc:492] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:54.989293 15691 tablet_bootstrap.cc:654] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:54.992409 15690 tablet_bootstrap.cc:492] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:54.992693 15690 ts_tablet_manager.cc:1367] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240503 14:15:54.994040 15690 raft_consensus.cc:348] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:54.994453 15690 raft_consensus.cc:374] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:54.994616 15690 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:54.995051 15690 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:54.996245 15690 ts_tablet_manager.cc:1398] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:54.997339 15690 tablet_bootstrap.cc:492] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:54.999166 15695 tablet_bootstrap.cc:492] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:54.999419 15695 ts_tablet_manager.cc:1367] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240503 14:15:55.000515 15695 raft_consensus.cc:348] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.000854 15695 raft_consensus.cc:374] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:55.001022 15695 raft_consensus.cc:729] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:55.001446 15695 consensus_queue.cc:260] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.002279 15695 ts_tablet_manager.cc:1398] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:55.002655 15695 tablet_bootstrap.cc:492] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:55.004868 15695 tablet_bootstrap.cc:654] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:55.006477 15690 tablet_bootstrap.cc:654] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:55.007884 15691 tablet_bootstrap.cc:492] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:55.008124 15691 ts_tablet_manager.cc:1367] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.013s
I20240503 14:15:55.009038 15691 raft_consensus.cc:348] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:55.009370 15691 raft_consensus.cc:374] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:55.009505 15691 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:55.009879 15691 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:55.010758 15691 ts_tablet_manager.cc:1398] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:55.011268 15691 tablet_bootstrap.cc:492] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:55.011385 15695 tablet_bootstrap.cc:492] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:55.011591 15695 ts_tablet_manager.cc:1367] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240503 14:15:55.012367 15695 raft_consensus.cc:348] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.012647 15695 raft_consensus.cc:374] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:55.012787 15695 raft_consensus.cc:729] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:55.012910 15691 tablet_bootstrap.cc:654] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:55.013134 15695 consensus_queue.cc:260] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.013705 15695 ts_tablet_manager.cc:1398] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:15:55.014987 15694 tablet_bootstrap.cc:492] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:55.015154 15694 ts_tablet_manager.cc:1367] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.018s
I20240503 14:15:55.016039 15694 raft_consensus.cc:348] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:55.016237 15694 raft_consensus.cc:374] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:55.016330 15694 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:55.016582 15694 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:55.017704 15694 ts_tablet_manager.cc:1398] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:15:55.018653 15690 tablet_bootstrap.cc:492] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:55.018868 15690 ts_tablet_manager.cc:1367] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240503 14:15:55.019964 15690 raft_consensus.cc:348] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.020283 15690 raft_consensus.cc:374] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:55.020440 15690 raft_consensus.cc:729] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:55.020797 15690 consensus_queue.cc:260] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.021620 15690 ts_tablet_manager.cc:1398] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:55.021998 15636 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:55.022147 15636 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:55.023033 15691 tablet_bootstrap.cc:492] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:55.023226 15691 ts_tablet_manager.cc:1367] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240503 14:15:55.024138 15636 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:15:55.024097 15691 raft_consensus.cc:348] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.024312 15691 raft_consensus.cc:374] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:55.024436 15691 raft_consensus.cc:729] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:55.024653 15691 consensus_queue.cc:260] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.025224 15691 ts_tablet_manager.cc:1398] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:55.028113 15618 raft_consensus.cc:482] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:55.028275 15618 raft_consensus.cc:504] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.030297 15618 leader_election.cc:288] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:55.036511 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:55.036775 14885 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:55.037395 14689 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:55.037892 15636 raft_consensus.cc:2781] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:55.038031 15636 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:55.038132 15636 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.040321 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:15:55.040817 14625 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:55.042536 15636 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:55.043617 15636 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:15:55.044239 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:55.044538 14885 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.048631 14885 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:15:55.049147 14689 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:55.049180 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:15:55.049420 14625 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.049705 15636 raft_consensus.cc:2781] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:55.053023 14625 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:15:55.058207 15636 raft_consensus.cc:686] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:15:55.058929 15636 consensus_queue.cc:237] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:55.068282 14481 catalog_manager.cc:5484] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:15:55.077687 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:15:55.077898 14755 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:15:55.078380 14820 leader_election.cc:302] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:55.078727 15618 raft_consensus.cc:2781] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:55.078841 15618 raft_consensus.cc:482] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:55.078934 15618 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.079113 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:15:55.079314 15015 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:15:55.080861 15618 raft_consensus.cc:504] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.081480 15618 leader_election.cc:288] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:55.081779 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:15:55.081916 14755 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.081905 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:15:55.082129 15015 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.084013 14755 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:15:55.084213 15015 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:15:55.084618 14819 leader_election.cc:302] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:15:55.084939 15618 raft_consensus.cc:2781] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:55.085642 15618 raft_consensus.cc:686] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:15:55.086346 15618 consensus_queue.cc:237] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.089346 14481 catalog_manager.cc:5484] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:15:55.102913 15636 raft_consensus.cc:482] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:55.103067 15636 raft_consensus.cc:504] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.103657 15636 leader_election.cc:288] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:55.104103 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:55.104355 14885 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:55.104758 14689 leader_election.cc:302] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:55.105039 15636 raft_consensus.cc:2781] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:55.105157 15636 raft_consensus.cc:482] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:55.105278 15636 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.107692 15636 raft_consensus.cc:504] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.108361 15636 leader_election.cc:288] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:55.108860 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:55.109042 14885 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.109997 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:15:55.110090 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:15:55.110280 15015 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:55.111352 14885 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:15:55.111706 14689 leader_election.cc:302] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:15:55.111984 15636 raft_consensus.cc:2781] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:55.112139 15636 raft_consensus.cc:686] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:15:55.112416 15636 consensus_queue.cc:237] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:15:55.114948 14477 catalog_manager.cc:5484] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:15:55.442220 15636 consensus_queue.cc:1035] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:55.448505 15636 consensus_queue.cc:1035] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:55.471866 15618 consensus_queue.cc:1035] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:55.476473 15618 consensus_queue.cc:1035] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:55.487612 15715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:55.487977 15715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:55.495498 15715 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:15:55.520259 15636 consensus_queue.cc:1035] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:55.523720 15015 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:55.526515 15636 consensus_queue.cc:1035] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 194.348 ms
time per row: 0.0971739 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:15:56.767431 14477 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:56.776399 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 46120f9b009a4e3d96143bd7cdca9f92 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.776671 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 46120f9b009a4e3d96143bd7cdca9f92 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.776891 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46120f9b009a4e3d96143bd7cdca9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.777122 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46120f9b009a4e3d96143bd7cdca9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.777057 14996 tablet_service.cc:1467] Processing CreateTablet for tablet d761bca383e1488491576b8af4cc2c3f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.777441 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761bca383e1488491576b8af4cc2c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.778371 14736 tablet_service.cc:1467] Processing CreateTablet for tablet 26c1a7f75f214911979a2fb26245c814 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.778765 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c1a7f75f214911979a2fb26245c814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.781155 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 46120f9b009a4e3d96143bd7cdca9f92 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.781596 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46120f9b009a4e3d96143bd7cdca9f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.787451 14603 tablet_service.cc:1467] Processing CreateTablet for tablet d761bca383e1488491576b8af4cc2c3f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.787451 14994 tablet_service.cc:1467] Processing CreateTablet for tablet 26c1a7f75f214911979a2fb26245c814 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.787843 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c1a7f75f214911979a2fb26245c814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.787851 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761bca383e1488491576b8af4cc2c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.790678 14866 tablet_service.cc:1467] Processing CreateTablet for tablet d761bca383e1488491576b8af4cc2c3f (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.791121 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d761bca383e1488491576b8af4cc2c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.791427 15779 tablet_bootstrap.cc:492] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:56.792238 15781 tablet_bootstrap.cc:492] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:56.793021 15779 tablet_bootstrap.cc:654] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.793756 15781 tablet_bootstrap.cc:654] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.795040 15779 tablet_bootstrap.cc:492] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:56.795167 15779 ts_tablet_manager.cc:1367] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:15:56.796115 15779 raft_consensus.cc:348] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.796366 15779 raft_consensus.cc:374] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.796586 15779 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.796864 15779 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.797545 15779 ts_tablet_manager.cc:1398] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:56.798719 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 26c1a7f75f214911979a2fb26245c814 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1632492018374dd092e3a8788d03e303]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:56.799063 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26c1a7f75f214911979a2fb26245c814. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:56.801146 15780 tablet_bootstrap.cc:492] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:56.803439 15780 tablet_bootstrap.cc:654] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.805222 15779 tablet_bootstrap.cc:492] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:56.805490 15781 tablet_bootstrap.cc:492] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:56.805637 15781 ts_tablet_manager.cc:1367] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240503 14:15:56.806375 15781 raft_consensus.cc:348] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.806574 15781 raft_consensus.cc:374] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.806650 15781 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.806926 15781 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.807417 15779 tablet_bootstrap.cc:654] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.807986 15765 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:56.808156 15765 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.808697 15765 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:56.809263 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:15:56.809960 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
W20240503 14:15:56.810578 14689 leader_election.cc:341] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:15:56.811164 14689 leader_election.cc:341] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9d4c39af431443189e8475561596810 (127.20.240.65:32989): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:15:56.811282 14689 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:15:56.811710 15765 raft_consensus.cc:2726] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:56.808035 15781 ts_tablet_manager.cc:1398] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:56.812741 15781 tablet_bootstrap.cc:492] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:56.814229 15781 tablet_bootstrap.cc:654] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.815397 15780 tablet_bootstrap.cc:492] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:56.815652 15780 ts_tablet_manager.cc:1367] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.010s
I20240503 14:15:56.816571 15779 tablet_bootstrap.cc:492] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:56.816538 15780 raft_consensus.cc:348] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.816768 15779 ts_tablet_manager.cc:1367] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240503 14:15:56.816816 15780 raft_consensus.cc:374] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.817005 15780 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.817301 15780 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.817507 15779 raft_consensus.cc:348] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.817708 15779 raft_consensus.cc:374] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.817793 15779 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.818069 15779 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.818887 15779 ts_tablet_manager.cc:1398] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:56.821007 15779 tablet_bootstrap.cc:492] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:56.821347 15786 tablet_bootstrap.cc:492] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:56.822479 15780 ts_tablet_manager.cc:1398] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20240503 14:15:56.822929 15781 tablet_bootstrap.cc:492] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:56.823113 15786 tablet_bootstrap.cc:654] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.823500 15781 ts_tablet_manager.cc:1367] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240503 14:15:56.824323 15779 tablet_bootstrap.cc:654] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.824301 15781 raft_consensus.cc:348] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.824658 15781 raft_consensus.cc:374] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.824786 15781 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.824795 15780 tablet_bootstrap.cc:492] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:56.825194 15781 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.825803 15781 ts_tablet_manager.cc:1398] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:56.828294 15780 tablet_bootstrap.cc:654] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.831689 15786 tablet_bootstrap.cc:492] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:56.831772 15780 tablet_bootstrap.cc:492] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:56.831799 15779 tablet_bootstrap.cc:492] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:56.831914 15786 ts_tablet_manager.cc:1367] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240503 14:15:56.831924 15779 ts_tablet_manager.cc:1367] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240503 14:15:56.831972 15780 ts_tablet_manager.cc:1367] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240503 14:15:56.832615 15786 raft_consensus.cc:348] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.832640 15779 raft_consensus.cc:348] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.832780 15786 raft_consensus.cc:374] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.832698 15780 raft_consensus.cc:348] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.832870 15786 raft_consensus.cc:729] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.832895 15780 raft_consensus.cc:374] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.832907 15779 raft_consensus.cc:374] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.832966 15780 raft_consensus.cc:729] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.832993 15779 raft_consensus.cc:729] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.833132 15786 consensus_queue.cc:260] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.833209 15779 consensus_queue.cc:260] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.833223 15780 consensus_queue.cc:260] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.833693 15786 ts_tablet_manager.cc:1398] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:56.833698 15780 ts_tablet_manager.cc:1398] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:15:56.833981 15780 tablet_bootstrap.cc:492] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:56.834820 15779 ts_tablet_manager.cc:1398] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:15:56.836325 15780 tablet_bootstrap.cc:654] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:56.838613 15780 tablet_bootstrap.cc:492] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:56.838778 15780 ts_tablet_manager.cc:1367] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 14:15:56.839427 15780 raft_consensus.cc:348] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.839581 15780 raft_consensus.cc:374] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:56.839644 15780 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:56.839815 15780 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.840256 15780 ts_tablet_manager.cc:1398] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:15:56.853827 15788 raft_consensus.cc:482] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:56.853972 15788 raft_consensus.cc:504] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.854544 15774 raft_consensus.cc:482] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:56.854697 15774 raft_consensus.cc:504] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.855382 15774 leader_election.cc:288] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:15:56.855861 15788 leader_election.cc:288] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:15:56.855829 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:56.855866 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:15:56.856043 14885 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:56.856181 15015 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:15:56.856585 14558 leader_election.cc:302] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:56.857038 15774 raft_consensus.cc:2781] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:56.857201 15774 raft_consensus.cc:482] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:56.857309 15774 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.861712 15774 raft_consensus.cc:504] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.863072 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:15:56.863345 15015 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.865279 15788 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:56.865489 15788 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.867086 15788 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:56.867341 15015 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:15:56.867861 14558 leader_election.cc:302] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:56.868252 15628 raft_consensus.cc:2781] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:56.872053 15774 leader_election.cc:288] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:15:56.878620 15628 raft_consensus.cc:686] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:15:56.879082 15628 consensus_queue.cc:237] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:56.884109 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:56.884352 14885 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.887416 14885 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:15:56.890503 14477 catalog_manager.cc:5484] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:15:56.890791 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:15:56.891036 14755 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:15:56.891530 14950 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:56.891968 15788 raft_consensus.cc:2781] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:56.892081 15788 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:56.892051 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:56.892195 15788 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.892302 14885 raft_consensus.cc:2379] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd in current term 1: Already voted for candidate e9d4c39af431443189e8475561596810 in this term.
I20240503 14:15:56.894973 15788 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.895498 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:15:56.895612 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:15:56.895833 14624 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:15:56.895975 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:15:56.895992 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:15:56.896171 14755 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.896193 14623 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.898058 15788 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:56.898377 14949 leader_election.cc:302] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810
I20240503 14:15:56.898873 14755 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:15:56.899016 14623 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:15:56.899166 15788 raft_consensus.cc:2726] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:15:56.899303 14950 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:56.899623 15788 raft_consensus.cc:2781] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:56.900036 15788 raft_consensus.cc:686] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:15:56.900435 15788 consensus_queue.cc:237] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.902477 15788 raft_consensus.cc:482] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:56.902663 15788 raft_consensus.cc:504] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.903167 15788 leader_election.cc:288] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:56.903524 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:15:56.903430 14477 catalog_manager.cc:5484] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:15:56.903728 14623 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:15:56.904052 14949 leader_election.cc:302] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:56.904315 15788 raft_consensus.cc:2781] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:56.904397 15788 raft_consensus.cc:482] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:56.904501 15788 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.904520 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:15:56.904742 14755 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:15:56.907153 15788 raft_consensus.cc:504] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.907685 15788 leader_election.cc:288] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:15:56.907965 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:15:56.908182 14623 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.908120 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:15:56.908313 14755 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:56.910872 14623 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:15:56.910943 14755 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:15:56.911288 14950 leader_election.cc:302] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:56.911522 15788 raft_consensus.cc:2781] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:56.911672 15788 raft_consensus.cc:686] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:15:56.911896 15788 consensus_queue.cc:237] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:56.914083 14477 catalog_manager.cc:5484] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
W20240503 14:15:57.239252 15804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:57.239542 15804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:57.246865 15804 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:15:57.342430 15788 consensus_queue.cc:1035] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:57.384122 15801 consensus_queue.cc:1035] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:57.386473 14885 raft_consensus.cc:1261] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:57.386587 15015 raft_consensus.cc:1261] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:57.387369 15836 consensus_queue.cc:1035] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:57.387703 15628 consensus_queue.cc:1035] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:57.388453 14756 raft_consensus.cc:1261] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:57.388839 14624 raft_consensus.cc:1261] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:57.389060 15788 consensus_queue.cc:1035] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:57.389590 15788 consensus_queue.cc:1035] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:57.420560 15842 mvcc.cc:205] Tried to move back new op lower bound from 7023598622248599552 to 7023598620312948736. Current Snapshot: MvccSnapshot[applied={T|T < 7023598622248599552}]
I20240503 14:15:57.421067 15839 mvcc.cc:205] Tried to move back new op lower bound from 7023598622248599552 to 7023598620312948736. Current Snapshot: MvccSnapshot[applied={T|T < 7023598622248599552}]
I20240503 14:15:57.482730 15834 mvcc.cc:205] Tried to move back new op lower bound from 7023598622282649600 to 7023598620312948736. Current Snapshot: MvccSnapshot[applied={T|T < 7023598622248599552}]
I20240503 14:15:57.520988 15843 mvcc.cc:205] Tried to move back new op lower bound from 7023598622280790016 to 7023598620180135936. Current Snapshot: MvccSnapshot[applied={T|T < 7023598622203920384}]
INSERT report
rows total: 2000
time total: 242.255 ms
time per row: 0.121127 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:15:58.606093 14477 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:58.615443 14866 tablet_service.cc:1467] Processing CreateTablet for tablet 34c5b61b82b34beb8d10677e24e6f418 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.615921 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c5b61b82b34beb8d10677e24e6f418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.615896 14603 tablet_service.cc:1467] Processing CreateTablet for tablet 34c5b61b82b34beb8d10677e24e6f418 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.615973 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 34c5b61b82b34beb8d10677e24e6f418 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.616300 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c5b61b82b34beb8d10677e24e6f418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.616451 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34c5b61b82b34beb8d10677e24e6f418. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.617633 14736 tablet_service.cc:1467] Processing CreateTablet for tablet 0184718c9fac4009b834726d661642f3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.617996 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0184718c9fac4009b834726d661642f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.618888 14605 tablet_service.cc:1467] Processing CreateTablet for tablet d24bc6dd77904c7e9580939b27641500 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.619274 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24bc6dd77904c7e9580939b27641500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.619252 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 0184718c9fac4009b834726d661642f3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.619645 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0184718c9fac4009b834726d661642f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.622915 14996 tablet_service.cc:1467] Processing CreateTablet for tablet d24bc6dd77904c7e9580939b27641500 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.623247 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24bc6dd77904c7e9580939b27641500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.627187 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 0184718c9fac4009b834726d661642f3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.627672 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0184718c9fac4009b834726d661642f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.630700 14864 tablet_service.cc:1467] Processing CreateTablet for tablet d24bc6dd77904c7e9580939b27641500 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5b666d44e2904d03978b4cb9e458932a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:58.631135 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d24bc6dd77904c7e9580939b27641500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:58.631934 15869 tablet_bootstrap.cc:492] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:58.633519 15869 tablet_bootstrap.cc:654] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.635924 15869 tablet_bootstrap.cc:492] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:58.636123 15869 ts_tablet_manager.cc:1367] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 14:15:58.636876 15869 raft_consensus.cc:348] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.637120 15869 raft_consensus.cc:374] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.637245 15869 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.637504 15869 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.638116 15869 ts_tablet_manager.cc:1398] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:15:58.638437 15869 tablet_bootstrap.cc:492] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:58.640062 15869 tablet_bootstrap.cc:654] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.649665 15872 tablet_bootstrap.cc:492] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:58.651682 15869 tablet_bootstrap.cc:492] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:58.651911 15869 ts_tablet_manager.cc:1367] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240503 14:15:58.652812 15869 raft_consensus.cc:348] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.653075 15872 tablet_bootstrap.cc:654] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.653230 15869 raft_consensus.cc:374] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.653398 15869 raft_consensus.cc:729] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.653821 15869 consensus_queue.cc:260] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.654742 15869 ts_tablet_manager.cc:1398] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:58.657529 15874 tablet_bootstrap.cc:492] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:58.658550 15870 tablet_bootstrap.cc:492] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:58.660295 15870 tablet_bootstrap.cc:654] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.660396 15874 tablet_bootstrap.cc:654] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.661085 15872 tablet_bootstrap.cc:492] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:58.661222 15872 ts_tablet_manager.cc:1367] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20240503 14:15:58.661820 15872 raft_consensus.cc:348] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.661981 15872 raft_consensus.cc:374] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.662075 15872 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.662129 15870 tablet_bootstrap.cc:492] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:58.662284 15870 ts_tablet_manager.cc:1367] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 14:15:58.662322 15872 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.662933 15872 ts_tablet_manager.cc:1398] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:58.663223 15872 tablet_bootstrap.cc:492] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:58.663081 15870 raft_consensus.cc:348] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.663872 15870 raft_consensus.cc:374] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.664052 15870 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.664398 15870 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.665040 15872 tablet_bootstrap.cc:654] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.668329 15870 ts_tablet_manager.cc:1398] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:58.669368 15870 tablet_bootstrap.cc:492] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:58.669916 15872 tablet_bootstrap.cc:492] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:58.670121 15872 ts_tablet_manager.cc:1367] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240503 14:15:58.670934 15872 raft_consensus.cc:348] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.671201 15872 raft_consensus.cc:374] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.671330 15872 raft_consensus.cc:729] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.671425 15870 tablet_bootstrap.cc:654] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.671650 15872 consensus_queue.cc:260] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.672289 15872 ts_tablet_manager.cc:1398] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:58.672626 15872 tablet_bootstrap.cc:492] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:58.674157 15872 tablet_bootstrap.cc:654] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.675573 15874 tablet_bootstrap.cc:492] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:58.675802 15874 ts_tablet_manager.cc:1367] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20240503 14:15:58.676792 15874 raft_consensus.cc:348] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.677064 15874 raft_consensus.cc:374] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.677174 15874 raft_consensus.cc:729] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.677552 15872 tablet_bootstrap.cc:492] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:58.677413 15874 consensus_queue.cc:260] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.677680 15872 ts_tablet_manager.cc:1367] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 14:15:58.678380 15872 raft_consensus.cc:348] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.678552 15872 raft_consensus.cc:374] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.678653 15872 raft_consensus.cc:729] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.678735 15874 ts_tablet_manager.cc:1398] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:15:58.678859 15872 consensus_queue.cc:260] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.678980 15874 tablet_bootstrap.cc:492] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:58.679396 15872 ts_tablet_manager.cc:1398] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:58.680341 15870 tablet_bootstrap.cc:492] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:58.680400 15874 tablet_bootstrap.cc:654] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:58.680506 15870 ts_tablet_manager.cc:1367] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240503 14:15:58.681257 15870 raft_consensus.cc:348] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.681488 15870 raft_consensus.cc:374] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.681581 15870 raft_consensus.cc:729] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.681862 15870 consensus_queue.cc:260] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.681998 15874 tablet_bootstrap.cc:492] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:58.682284 15874 ts_tablet_manager.cc:1367] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:15:58.682478 15870 ts_tablet_manager.cc:1398] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:15:58.683841 15874 raft_consensus.cc:348] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.684067 15874 raft_consensus.cc:374] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:58.684150 15874 raft_consensus.cc:729] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:58.684366 15874 consensus_queue.cc:260] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.684845 15874 ts_tablet_manager.cc:1398] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:58.792677 15765 raft_consensus.cc:482] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:58.792830 15765 raft_consensus.cc:504] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.793344 15765 leader_election.cc:288] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:58.793855 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:15:58.793833 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:58.794077 15015 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:58.794099 14885 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:58.794517 14689 leader_election.cc:302] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:58.794780 15765 raft_consensus.cc:2781] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:58.794904 15765 raft_consensus.cc:482] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:58.794999 15765 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.796907 15765 raft_consensus.cc:504] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.797547 15765 leader_election.cc:288] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:15:58.798058 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:58.798074 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:15:58.798285 14885 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.798301 15015 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.800431 15015 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:15:58.800850 14689 leader_election.cc:302] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:58.801012 14885 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:15:58.801160 15765 raft_consensus.cc:2781] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:58.801780 15765 raft_consensus.cc:686] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:15:58.802086 15765 consensus_queue.cc:237] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.804885 14476 catalog_manager.cc:5484] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:15:58.837036 15765 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:58.837179 15765 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.837741 15765 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:58.838130 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:58.838259 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:15:58.838415 14885 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:58.838487 14623 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:15:58.838800 14689 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:58.839090 15765 raft_consensus.cc:2781] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:58.839169 15765 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:58.839259 15765 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.840965 15765 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.841403 15765 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:58.841780 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:58.841835 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:15:58.841928 14885 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.841984 14623 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.843765 14623 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:15:58.844059 14885 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:15:58.844122 14689 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:15:58.844372 15765 raft_consensus.cc:2781] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:58.844911 15765 raft_consensus.cc:686] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:15:58.845172 15765 consensus_queue.cc:237] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:15:58.847676 14476 catalog_manager.cc:5484] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:15:58.888322 15801 raft_consensus.cc:482] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:58.888520 15801 raft_consensus.cc:504] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.889021 15801 leader_election.cc:288] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:58.889503 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:15:58.889756 14885 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:15:58.889675 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:15:58.889937 14623 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:15:58.890172 14949 leader_election.cc:302] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:15:58.890480 15801 raft_consensus.cc:2781] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:58.890564 15801 raft_consensus.cc:482] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:58.890666 15801 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.892443 15801 raft_consensus.cc:504] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.892905 15801 leader_election.cc:288] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:58.893303 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:15:58.893354 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:15:58.893533 14885 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.893554 14623 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:58.895677 14885 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:15:58.895794 14623 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:15:58.896034 14949 leader_election.cc:302] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:15:58.896265 15801 raft_consensus.cc:2781] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:58.896656 15801 raft_consensus.cc:686] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:15:58.896925 15801 consensus_queue.cc:237] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:58.899350 14476 catalog_manager.cc:5484] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:15:59.209604 15765 consensus_queue.cc:1035] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:59.215167 15765 consensus_queue.cc:1035] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:15:59.241480 15888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:15:59.241748 15888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:15:59.249150 15888 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:15:59.287447 15765 consensus_queue.cc:1035] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:59.292876 15884 consensus_queue.cc:1035] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:59.400664 14885 raft_consensus.cc:1261] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:59.400662 14623 raft_consensus.cc:1261] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:15:59.401357 15801 consensus_queue.cc:1035] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:15:59.401897 15801 consensus_queue.cc:1035] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 232.333 ms
time per row: 0.116167 ms
I20240503 14:15:59.634764 14480 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:15:59.642984 14736 tablet_service.cc:1467] Processing CreateTablet for tablet 5fbeca22e62946dd922046cf5cdaef0c (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.643133 14864 tablet_service.cc:1467] Processing CreateTablet for tablet 5fbeca22e62946dd922046cf5cdaef0c (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.643462 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbeca22e62946dd922046cf5cdaef0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.643560 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbeca22e62946dd922046cf5cdaef0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.643478 14603 tablet_service.cc:1467] Processing CreateTablet for tablet 5fbeca22e62946dd922046cf5cdaef0c (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.643850 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbeca22e62946dd922046cf5cdaef0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.645164 14996 tablet_service.cc:1467] Processing CreateTablet for tablet 340fdb957a4349c29a45e66ddb303104 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.645581 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340fdb957a4349c29a45e66ddb303104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.647241 14735 tablet_service.cc:1467] Processing CreateTablet for tablet 340fdb957a4349c29a45e66ddb303104 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.647578 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340fdb957a4349c29a45e66ddb303104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.649569 14865 tablet_service.cc:1467] Processing CreateTablet for tablet c97701fcd6674dadbb15881115dbc4f5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.649917 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97701fcd6674dadbb15881115dbc4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.650558 14734 tablet_service.cc:1467] Processing CreateTablet for tablet c97701fcd6674dadbb15881115dbc4f5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.650669 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 340fdb957a4349c29a45e66ddb303104 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.650892 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97701fcd6674dadbb15881115dbc4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.650991 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 340fdb957a4349c29a45e66ddb303104. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.651818 14995 tablet_service.cc:1467] Processing CreateTablet for tablet c97701fcd6674dadbb15881115dbc4f5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=f0c491a783e649429d0a1e93e50ad7f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:15:59.652137 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97701fcd6674dadbb15881115dbc4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:15:59.661689 15954 tablet_bootstrap.cc:492] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:59.662175 15952 tablet_bootstrap.cc:492] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:59.662976 15953 tablet_bootstrap.cc:492] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:59.664026 15954 tablet_bootstrap.cc:654] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.664386 15952 tablet_bootstrap.cc:654] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.664680 15953 tablet_bootstrap.cc:654] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.669559 15951 tablet_bootstrap.cc:492] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:59.671317 15951 tablet_bootstrap.cc:654] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.671523 15954 tablet_bootstrap.cc:492] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:59.671717 15954 ts_tablet_manager.cc:1367] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240503 14:15:59.672582 15954 raft_consensus.cc:348] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.672868 15953 tablet_bootstrap.cc:492] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:59.672914 15954 raft_consensus.cc:374] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.673070 15954 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.673091 15953 ts_tablet_manager.cc:1367] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 14:15:59.673386 15954 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.674018 15953 raft_consensus.cc:348] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:59.674163 15954 ts_tablet_manager.cc:1398] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:15:59.674226 15953 raft_consensus.cc:374] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.674387 15953 raft_consensus.cc:729] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.674651 15953 consensus_queue.cc:260] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:59.674866 15952 tablet_bootstrap.cc:492] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:59.675022 15952 ts_tablet_manager.cc:1367] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240503 14:15:59.675401 15951 tablet_bootstrap.cc:492] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:59.675524 15951 ts_tablet_manager.cc:1367] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:15:59.675825 15952 raft_consensus.cc:348] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.676051 15952 raft_consensus.cc:374] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.676149 15952 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.676411 15953 ts_tablet_manager.cc:1398] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:15:59.676290 15951 raft_consensus.cc:348] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.676496 15951 raft_consensus.cc:374] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.676398 15952 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.676746 15953 tablet_bootstrap.cc:492] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:15:59.676949 15952 ts_tablet_manager.cc:1398] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:15:59.677632 15952 tablet_bootstrap.cc:492] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:15:59.677800 15954 tablet_bootstrap.cc:492] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:59.679184 15952 tablet_bootstrap.cc:654] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.679482 15954 tablet_bootstrap.cc:654] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.680652 15951 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.680935 15951 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.682101 15951 ts_tablet_manager.cc:1398] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:15:59.682382 15951 tablet_bootstrap.cc:492] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:15:59.682762 15953 tablet_bootstrap.cc:654] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.683960 15954 tablet_bootstrap.cc:492] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:59.684000 15951 tablet_bootstrap.cc:654] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.684172 15954 ts_tablet_manager.cc:1367] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:15:59.685405 15952 tablet_bootstrap.cc:492] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:15:59.685336 15954 raft_consensus.cc:348] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:59.685613 15952 ts_tablet_manager.cc:1367] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:15:59.685735 15954 raft_consensus.cc:374] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.685902 15954 raft_consensus.cc:729] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.686210 15954 consensus_queue.cc:260] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:59.686697 15952 raft_consensus.cc:348] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:59.687021 15952 raft_consensus.cc:374] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.687153 15952 raft_consensus.cc:729] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.687495 15954 ts_tablet_manager.cc:1398] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:15:59.687541 15952 consensus_queue.cc:260] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:15:59.687824 15954 tablet_bootstrap.cc:492] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:15:59.689095 15951 tablet_bootstrap.cc:492] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:15:59.690111 15954 tablet_bootstrap.cc:654] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:15:59.690171 15953 tablet_bootstrap.cc:492] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:15:59.690359 15953 ts_tablet_manager.cc:1367] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240503 14:15:59.690708 15952 ts_tablet_manager.cc:1398] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:15:59.690909 15951 ts_tablet_manager.cc:1367] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 14:15:59.691099 15953 raft_consensus.cc:348] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.691311 15953 raft_consensus.cc:374] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.691403 15953 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.691692 15953 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.691735 15951 raft_consensus.cc:348] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.691970 15951 raft_consensus.cc:374] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.692061 15951 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.692212 15953 ts_tablet_manager.cc:1398] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:15:59.692370 15951 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.693611 15951 ts_tablet_manager.cc:1398] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:15:59.694706 15954 tablet_bootstrap.cc:492] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:15:59.694842 15954 ts_tablet_manager.cc:1367] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240503 14:15:59.695525 15954 raft_consensus.cc:348] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.695691 15954 raft_consensus.cc:374] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:15:59.695785 15954 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:15:59.695986 15954 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.696873 15954 ts_tablet_manager.cc:1398] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:15:59.702121 15712 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:15:59.702287 15712 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.702977 15712 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:59.703236 14754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:15:59.703430 14754 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:15:59.703853 14820 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:59.704119 15712 raft_consensus.cc:2781] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:15:59.704218 15712 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:15:59.704308 15712 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:59.706663 15712 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.707362 15712 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:15:59.707667 14754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:15:59.707859 14754 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:15:59.709934 14754 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:15:59.710107 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:15:59.710173 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:15:59.710320 14820 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:15:59.710378 14623 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:15:59.710588 15712 raft_consensus.cc:2781] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:15:59.711048 15712 raft_consensus.cc:686] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:15:59.711597 15712 consensus_queue.cc:237] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:15:59.714187 14480 catalog_manager.cc:5484] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:00.013567 15928 raft_consensus.cc:482] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:00.013751 15928 raft_consensus.cc:504] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.014425 15928 leader_election.cc:288] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:00.014770 14754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:00.015002 14754 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:00.014991 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:16:00.015257 14885 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:00.015388 14950 leader_election.cc:302] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:00.015661 15928 raft_consensus.cc:2781] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:00.015799 15928 raft_consensus.cc:482] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:00.015904 15928 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.017423 15933 raft_consensus.cc:482] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:00.017567 15933 raft_consensus.cc:504] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:00.018014 15928 raft_consensus.cc:504] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.018345 15933 leader_election.cc:288] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:00.018636 15928 leader_election.cc:288] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:00.018821 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:00.018978 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:00.019037 14624 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:16:00.019129 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:00.019227 15016 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:16:00.019328 14885 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.019623 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:00.019804 14756 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.019929 14689 leader_election.cc:302] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:00.020450 15933 raft_consensus.cc:2781] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:00.020576 15933 raft_consensus.cc:482] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:00.020663 15933 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.021957 14756 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:00.022154 14885 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:00.022477 14950 leader_election.cc:302] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:00.022990 15968 raft_consensus.cc:2781] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:00.023150 15968 raft_consensus.cc:686] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:00.023444 15933 raft_consensus.cc:504] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:00.023679 15968 consensus_queue.cc:237] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.024327 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:00.024565 15016 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.026364 14480 catalog_manager.cc:5484] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:00.026857 15016 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:16:00.027393 14689 leader_election.cc:302] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:00.027472 15933 leader_election.cc:288] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:00.027653 15933 raft_consensus.cc:2781] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:00.027776 15933 raft_consensus.cc:686] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:00.028259 15933 consensus_queue.cc:237] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:00.030169 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:00.030398 14624 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.030984 14480 catalog_manager.cc:5484] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:00.032480 14624 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:16:00.130685 15712 consensus_queue.cc:1035] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:00.136191 14624 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.139106 15712 consensus_queue.cc:1035] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:00.439577 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:00.439859 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:16:00.441403 15970 consensus_queue.cc:1035] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:00.445729 15970 consensus_queue.cc:1035] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:00.449864 15972 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:16:00.478672 15968 consensus_queue.cc:1035] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:00.483162 15968 consensus_queue.cc:1035] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 232.776 ms
time per row: 0.116388 ms
I20240503 14:16:00.835608 14480 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:16:00.843348 14603 tablet_service.cc:1467] Processing CreateTablet for tablet c32b296d9dda42bd81f5e7aa1edc5774 (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.843786 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32b296d9dda42bd81f5e7aa1edc5774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.843962 14865 tablet_service.cc:1467] Processing CreateTablet for tablet c32b296d9dda42bd81f5e7aa1edc5774 (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.844439 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32b296d9dda42bd81f5e7aa1edc5774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.845054 14864 tablet_service.cc:1467] Processing CreateTablet for tablet 054ce7121a9f4881aa2d38615f0c04a1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.845140 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 054ce7121a9f4881aa2d38615f0c04a1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.845422 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054ce7121a9f4881aa2d38615f0c04a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.845574 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054ce7121a9f4881aa2d38615f0c04a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.846977 14866 tablet_service.cc:1467] Processing CreateTablet for tablet f145bdadec9d48dfb7370766e5196f4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.847355 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f145bdadec9d48dfb7370766e5196f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.850808 14604 tablet_service.cc:1467] Processing CreateTablet for tablet f145bdadec9d48dfb7370766e5196f4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.851152 14604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f145bdadec9d48dfb7370766e5196f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.853710 16037 tablet_bootstrap.cc:492] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:00.853722 14735 tablet_service.cc:1467] Processing CreateTablet for tablet f145bdadec9d48dfb7370766e5196f4f (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.854113 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f145bdadec9d48dfb7370766e5196f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.855477 16037 tablet_bootstrap.cc:654] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.859402 14996 tablet_service.cc:1467] Processing CreateTablet for tablet c32b296d9dda42bd81f5e7aa1edc5774 (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.859838 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32b296d9dda42bd81f5e7aa1edc5774. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.866123 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 054ce7121a9f4881aa2d38615f0c04a1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=26b211ba36b244e7b4f9c45685929d47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:00.866279 16040 tablet_bootstrap.cc:492] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:00.866518 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 054ce7121a9f4881aa2d38615f0c04a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:00.868340 16040 tablet_bootstrap.cc:654] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.874344 16042 tablet_bootstrap.cc:492] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:00.875087 16037 tablet_bootstrap.cc:492] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:00.875339 16037 ts_tablet_manager.cc:1367] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240503 14:16:00.876273 16037 raft_consensus.cc:348] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.876593 16037 raft_consensus.cc:374] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.876737 16037 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.877089 16037 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.877905 16037 ts_tablet_manager.cc:1398] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:16:00.880440 16042 tablet_bootstrap.cc:654] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.881579 16037 tablet_bootstrap.cc:492] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:00.885406 16037 tablet_bootstrap.cc:654] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.889487 16038 tablet_bootstrap.cc:492] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:00.889940 16042 tablet_bootstrap.cc:492] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:00.890040 16037 tablet_bootstrap.cc:492] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:00.890121 16042 ts_tablet_manager.cc:1367] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240503 14:16:00.890347 16037 ts_tablet_manager.cc:1367] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240503 14:16:00.890944 16042 raft_consensus.cc:348] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:00.891230 16042 raft_consensus.cc:374] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.891147 16037 raft_consensus.cc:348] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.891290 16038 tablet_bootstrap.cc:654] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.891359 16037 raft_consensus.cc:374] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.891409 16042 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.891498 16037 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.891619 16040 tablet_bootstrap.cc:492] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:00.891789 16040 ts_tablet_manager.cc:1367] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.011s
I20240503 14:16:00.891746 16042 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:00.891836 16037 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.892577 16040 raft_consensus.cc:348] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.892802 16040 raft_consensus.cc:374] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.892930 16040 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.893213 16037 ts_tablet_manager.cc:1398] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:00.893215 16040 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.893584 16037 tablet_bootstrap.cc:492] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:00.894132 16042 ts_tablet_manager.cc:1398] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:16:00.894615 16040 ts_tablet_manager.cc:1398] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:16:00.894937 16040 tablet_bootstrap.cc:492] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:00.896191 16037 tablet_bootstrap.cc:654] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.897292 16040 tablet_bootstrap.cc:654] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.899251 16038 tablet_bootstrap.cc:492] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:00.899457 16038 ts_tablet_manager.cc:1367] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240503 14:16:00.900219 16038 raft_consensus.cc:348] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.900380 16038 raft_consensus.cc:374] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.900506 16038 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.900738 16038 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.901347 16038 ts_tablet_manager.cc:1398] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:16:00.901654 16038 tablet_bootstrap.cc:492] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:00.903527 16038 tablet_bootstrap.cc:654] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.905282 16037 tablet_bootstrap.cc:492] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:00.905508 16037 ts_tablet_manager.cc:1367] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 14:16:00.906621 16037 raft_consensus.cc:348] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:00.906837 16037 raft_consensus.cc:374] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.906935 16037 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.907166 16037 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:00.907474 16038 tablet_bootstrap.cc:492] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:00.907546 16040 tablet_bootstrap.cc:492] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:00.907663 16038 ts_tablet_manager.cc:1367] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240503 14:16:00.907706 16040 ts_tablet_manager.cc:1367] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.010s
I20240503 14:16:00.908367 16038 raft_consensus.cc:348] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:00.908488 16037 ts_tablet_manager.cc:1398] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:00.908454 16040 raft_consensus.cc:348] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.908682 16040 raft_consensus.cc:374] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.908766 16040 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.909003 16040 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.909456 16040 ts_tablet_manager.cc:1398] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:16:00.909592 16038 raft_consensus.cc:374] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.909693 16038 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.909945 16038 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:00.911425 16038 ts_tablet_manager.cc:1398] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:16:00.911772 16038 tablet_bootstrap.cc:492] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:00.913378 16038 tablet_bootstrap.cc:654] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:00.915195 16038 tablet_bootstrap.cc:492] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:00.915364 16038 ts_tablet_manager.cc:1367] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:16:00.915885 16038 raft_consensus.cc:348] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.916025 16038 raft_consensus.cc:374] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:00.916117 16038 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:00.916316 16038 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:00.916709 16038 ts_tablet_manager.cc:1398] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:16:00.943168 15712 raft_consensus.cc:482] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:00.943328 15712 raft_consensus.cc:504] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.943879 15712 leader_election.cc:288] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:00.944345 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:00.944352 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:00.944532 15016 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:00.944545 14623 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:00.944943 14819 leader_election.cc:302] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:00.945261 15712 raft_consensus.cc:2781] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:00.945348 15712 raft_consensus.cc:482] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:00.945488 15712 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.947283 15712 raft_consensus.cc:504] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.947763 15712 leader_election.cc:288] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:00.948104 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:00.948215 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:00.948374 14623 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.948374 15016 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:00.951062 14623 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:00.951078 15016 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:00.951423 14819 leader_election.cc:302] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:00.951659 15712 raft_consensus.cc:2781] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:00.951840 15712 raft_consensus.cc:686] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:00.952073 15712 consensus_queue.cc:237] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:00.954721 14482 catalog_manager.cc:5484] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:01.180505 15851 raft_consensus.cc:482] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:01.180656 15851 raft_consensus.cc:504] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:01.181391 15851 leader_election.cc:288] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:01.181723 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:16:01.181967 14885 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:16:01.181916 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:01.182147 14755 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:16:01.182412 14558 leader_election.cc:302] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:01.182744 15851 raft_consensus.cc:2781] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:01.182859 15851 raft_consensus.cc:482] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:01.182930 15851 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:01.184801 15851 raft_consensus.cc:504] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:01.185318 15851 leader_election.cc:288] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:01.185662 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:01.185730 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:01.185843 14885 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:01.185943 14755 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:01.187832 14885 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:16:01.188031 14755 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:16:01.188261 14558 leader_election.cc:302] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:01.188530 15851 raft_consensus.cc:2781] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:01.188666 15851 raft_consensus.cc:686] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:01.188946 15851 consensus_queue.cc:237] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:01.191654 14482 catalog_manager.cc:5484] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:01.193856 15928 raft_consensus.cc:482] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:01.194044 15928 raft_consensus.cc:504] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:01.194685 15928 leader_election.cc:288] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:01.195057 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:01.195143 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:16:01.195247 14623 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:01.195372 14885 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:01.195660 14949 leader_election.cc:302] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:01.195948 15928 raft_consensus.cc:2781] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:01.196061 15928 raft_consensus.cc:482] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:01.196153 15928 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:01.197937 15928 raft_consensus.cc:504] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:01.198637 15928 leader_election.cc:288] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:01.198997 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:01.199164 14623 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:01.199110 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:01.199301 14885 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:01.201083 14885 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:01.201402 14949 leader_election.cc:302] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:01.201704 15928 raft_consensus.cc:2781] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:01.201864 15928 raft_consensus.cc:686] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:01.202142 15928 consensus_queue.cc:237] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:01.204891 14482 catalog_manager.cc:5484] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:01.228121 14623 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:01.454149 15712 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:01.461580 16056 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:01.575930 16054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:01.576224 16054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:16:01.582119 15851 consensus_queue.cc:1035] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:01.584761 16054 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:16:01.586373 15851 consensus_queue.cc:1035] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
I20240503 14:16:01.635593 15928 consensus_queue.cc:1035] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:01.641588 16030 consensus_queue.cc:1035] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 301.194 ms
time per row: 0.150597 ms
I20240503 14:16:02.027256 14479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:16:02.035693 14605 tablet_service.cc:1467] Processing CreateTablet for tablet aab2e8db77094d73948bed276f95f8d3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.036176 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab2e8db77094d73948bed276f95f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.036082 14996 tablet_service.cc:1467] Processing CreateTablet for tablet aab2e8db77094d73948bed276f95f8d3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.036501 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab2e8db77094d73948bed276f95f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.037088 14735 tablet_service.cc:1467] Processing CreateTablet for tablet aab2e8db77094d73948bed276f95f8d3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.037292 14734 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceccdb68dda4765ad1c8e4051d2e4c1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.037495 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aab2e8db77094d73948bed276f95f8d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.037739 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceccdb68dda4765ad1c8e4051d2e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.039744 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0cc940dc9e4511856fecf6e43028e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.039773 14994 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceccdb68dda4765ad1c8e4051d2e4c1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.040082 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0cc940dc9e4511856fecf6e43028e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.040406 14736 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0cc940dc9e4511856fecf6e43028e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.040752 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0cc940dc9e4511856fecf6e43028e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.043584 14866 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0cc940dc9e4511856fecf6e43028e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.043994 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0cc940dc9e4511856fecf6e43028e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.044121 14603 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceccdb68dda4765ad1c8e4051d2e4c1 (DEFAULT_TABLE table=rebalance_test_table_6 [id=82c0e09898a24e78af5bdb0dc4f00e71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:02.044507 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceccdb68dda4765ad1c8e4051d2e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.046542 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceccdb68dda4765ad1c8e4051d2e4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:02.053469 16120 tablet_bootstrap.cc:492] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:02.054190 16121 tablet_bootstrap.cc:492] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:02.054922 16122 tablet_bootstrap.cc:492] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:02.055193 16120 tablet_bootstrap.cc:654] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.056317 16121 tablet_bootstrap.cc:654] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.057020 16122 tablet_bootstrap.cc:654] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.065641 16126 tablet_bootstrap.cc:492] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:02.065641 16122 tablet_bootstrap.cc:492] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:02.065773 16122 ts_tablet_manager.cc:1367] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 14:16:02.066668 16122 raft_consensus.cc:348] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.067013 16122 raft_consensus.cc:374] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.067149 16122 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.067219 16121 tablet_bootstrap.cc:492] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:02.067409 16120 tablet_bootstrap.cc:492] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:02.067438 16121 ts_tablet_manager.cc:1367] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:16:02.067595 16120 ts_tablet_manager.cc:1367] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20240503 14:16:02.067461 16122 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.067843 16126 tablet_bootstrap.cc:654] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.068264 16122 ts_tablet_manager.cc:1398] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:02.068477 16120 raft_consensus.cc:348] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.069101 16121 raft_consensus.cc:348] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.069300 16121 raft_consensus.cc:374] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.069435 16121 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.069665 16121 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.070171 16121 ts_tablet_manager.cc:1398] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:16:02.070453 16121 tablet_bootstrap.cc:492] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:02.071978 16121 tablet_bootstrap.cc:654] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.075706 16120 raft_consensus.cc:374] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.075917 16120 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.076364 16126 tablet_bootstrap.cc:492] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:02.076315 16120 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.076581 16126 ts_tablet_manager.cc:1367] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 14:16:02.077394 16126 raft_consensus.cc:348] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.077555 16126 raft_consensus.cc:374] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.077682 16126 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.077901 16126 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.078305 16120 ts_tablet_manager.cc:1398] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240503 14:16:02.078476 16126 ts_tablet_manager.cc:1398] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:16:02.078569 16120 tablet_bootstrap.cc:492] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:02.078775 16126 tablet_bootstrap.cc:492] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:02.080224 16126 tablet_bootstrap.cc:654] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.081375 16120 tablet_bootstrap.cc:654] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.083490 16120 tablet_bootstrap.cc:492] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:02.083619 16126 tablet_bootstrap.cc:492] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:02.083635 16120 ts_tablet_manager.cc:1367] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:16:02.083770 16126 ts_tablet_manager.cc:1367] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:16:02.084138 16121 tablet_bootstrap.cc:492] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:02.084278 16121 ts_tablet_manager.cc:1367] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20240503 14:16:02.084357 16120 raft_consensus.cc:348] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.084442 16126 raft_consensus.cc:348] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.084578 16120 raft_consensus.cc:374] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.084636 16126 raft_consensus.cc:374] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.084647 16120 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.084717 16126 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.084954 16126 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.084960 16120 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.084976 16121 raft_consensus.cc:348] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.085177 16121 raft_consensus.cc:374] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.085274 16121 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.085537 16120 ts_tablet_manager.cc:1398] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:16:02.085552 16126 ts_tablet_manager.cc:1398] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:02.085471 16121 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.085834 16120 tablet_bootstrap.cc:492] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:02.086020 16121 ts_tablet_manager.cc:1398] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:16:02.086319 16121 tablet_bootstrap.cc:492] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:02.087148 16120 tablet_bootstrap.cc:654] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.087944 16121 tablet_bootstrap.cc:654] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:02.094727 16120 tablet_bootstrap.cc:492] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:02.094727 16121 tablet_bootstrap.cc:492] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:02.094868 16120 ts_tablet_manager.cc:1367] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:16:02.094867 16121 ts_tablet_manager.cc:1367] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:16:02.095559 16121 raft_consensus.cc:348] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.095559 16120 raft_consensus.cc:348] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.095760 16121 raft_consensus.cc:374] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.095760 16120 raft_consensus.cc:374] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:02.095850 16120 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.095849 16121 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:02.096076 16120 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.096065 16121 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.096603 16121 ts_tablet_manager.cc:1398] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:16:02.096627 16120 ts_tablet_manager.cc:1398] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:02.104874 16056 raft_consensus.cc:482] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:02.105038 16056 raft_consensus.cc:504] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.105767 16056 leader_election.cc:288] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:02.106228 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:02.106312 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:02.106434 14755 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:02.106539 15015 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:02.106778 14820 leader_election.cc:302] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:02.107041 16056 raft_consensus.cc:2781] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:02.107143 16056 raft_consensus.cc:482] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:02.107245 16056 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.109006 16056 raft_consensus.cc:504] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.109560 16056 leader_election.cc:288] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:02.109892 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:02.110073 14755 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.110005 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:02.110213 15015 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.112876 15015 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:02.112876 14755 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:02.113268 15970 raft_consensus.cc:482] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:02.113301 14819 leader_election.cc:302] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:02.113413 15970 raft_consensus.cc:504] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.113655 16056 raft_consensus.cc:2781] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:02.113829 16056 raft_consensus.cc:686] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:02.114130 15970 leader_election.cc:288] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:02.114070 16056 consensus_queue.cc:237] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.114778 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:02.114810 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:02.114975 15015 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:16:02.114990 14624 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:16:02.115361 14689 leader_election.cc:302] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:02.115842 15970 raft_consensus.cc:2781] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:02.115975 15970 raft_consensus.cc:482] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:02.116101 15970 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.117341 14476 catalog_manager.cc:5484] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:02.118139 15970 raft_consensus.cc:504] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.118817 15970 leader_election.cc:288] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:02.119292 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:02.119464 15015 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.119482 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:02.119683 14624 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.121954 15015 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:16:02.122015 14624 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:16:02.122347 14689 leader_election.cc:302] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:02.123162 16133 raft_consensus.cc:2781] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:02.123322 16133 raft_consensus.cc:686] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:02.123606 16133 consensus_queue.cc:237] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:02.125844 14481 catalog_manager.cc:5484] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:02.179268 15928 raft_consensus.cc:482] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:02.179450 15928 raft_consensus.cc:504] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.180099 15928 leader_election.cc:288] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:02.180526 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:02.180598 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:02.180753 14623 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:02.180893 14755 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:02.181208 14949 leader_election.cc:302] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:02.181476 15928 raft_consensus.cc:2781] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:02.181605 15928 raft_consensus.cc:482] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:02.181680 15928 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.183558 15928 raft_consensus.cc:504] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.184058 15928 leader_election.cc:288] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:02.184417 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:02.184469 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:02.184613 14623 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.184634 14755 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:02.186553 14755 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:02.186547 14623 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:02.186923 14949 leader_election.cc:302] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:02.187192 15928 raft_consensus.cc:2781] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:02.187350 15928 raft_consensus.cc:686] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:02.187610 15928 consensus_queue.cc:237] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:02.190343 14481 catalog_manager.cc:5484] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:02.531395 15970 consensus_queue.cc:1035] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:02.531320 16138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:02.531651 16138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:16:02.536120 16056 consensus_queue.cc:1035] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:02.538563 16133 consensus_queue.cc:1035] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:02.547297 16138 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:16:02.559374 15712 consensus_queue.cc:1035] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240503 14:16:02.715415 14755 raft_consensus.cc:1261] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:16:02.716104 15928 consensus_queue.cc:1035] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:02.716590 14623 raft_consensus.cc:1261] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:16:02.717304 15928 consensus_queue.cc:1035] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 276.671 ms
time per row: 0.138335 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:16:03.916754 14479 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:16:03.925812 14866 tablet_service.cc:1467] Processing CreateTablet for tablet 46d6b19383c449198147cf7a999199d1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.926316 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d6b19383c449198147cf7a999199d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.926225 14603 tablet_service.cc:1467] Processing CreateTablet for tablet 46d6b19383c449198147cf7a999199d1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.926649 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d6b19383c449198147cf7a999199d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.928124 14864 tablet_service.cc:1467] Processing CreateTablet for tablet 843132c2c55645d598035edd0d2233c3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.928170 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 843132c2c55645d598035edd0d2233c3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.928489 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843132c2c55645d598035edd0d2233c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.928575 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843132c2c55645d598035edd0d2233c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.930656 14994 tablet_service.cc:1467] Processing CreateTablet for tablet b6b42a4b52d94e1e9c72cdc2ab70aaca (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.930989 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b42a4b52d94e1e9c72cdc2ab70aaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.930970 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 843132c2c55645d598035edd0d2233c3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.931303 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 843132c2c55645d598035edd0d2233c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.934695 14865 tablet_service.cc:1467] Processing CreateTablet for tablet b6b42a4b52d94e1e9c72cdc2ab70aaca (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.935192 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b42a4b52d94e1e9c72cdc2ab70aaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.939319 14736 tablet_service.cc:1467] Processing CreateTablet for tablet 46d6b19383c449198147cf7a999199d1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.939772 14734 tablet_service.cc:1467] Processing CreateTablet for tablet b6b42a4b52d94e1e9c72cdc2ab70aaca (DEFAULT_TABLE table=rebalance_test_table_7 [id=6d4d9de9495c48cd950bf72fd786ad5e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:03.939780 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d6b19383c449198147cf7a999199d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.940429 16208 tablet_bootstrap.cc:492] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:03.942618 16208 tablet_bootstrap.cc:654] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.946532 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b42a4b52d94e1e9c72cdc2ab70aaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:03.955565 16211 tablet_bootstrap.cc:492] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:03.958777 16211 tablet_bootstrap.cc:654] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.966387 16213 tablet_bootstrap.cc:492] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:03.967084 16209 tablet_bootstrap.cc:492] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:03.967782 16211 tablet_bootstrap.cc:492] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:03.968044 16211 ts_tablet_manager.cc:1367] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240503 14:16:03.968609 16208 tablet_bootstrap.cc:492] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:03.968761 16208 ts_tablet_manager.cc:1367] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.013s
I20240503 14:16:03.968947 16209 tablet_bootstrap.cc:654] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.969034 16211 raft_consensus.cc:348] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:03.969327 16211 raft_consensus.cc:374] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.969462 16211 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.969494 16208 raft_consensus.cc:348] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:03.969695 16208 raft_consensus.cc:374] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.969786 16208 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.970026 16208 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:03.970003 16211 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:03.971909 16208 ts_tablet_manager.cc:1398] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:16:03.972223 16208 tablet_bootstrap.cc:492] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:03.973253 16213 tablet_bootstrap.cc:654] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.973958 16208 tablet_bootstrap.cc:654] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.977582 16211 ts_tablet_manager.cc:1398] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:16:03.979516 16209 tablet_bootstrap.cc:492] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:03.979720 16209 ts_tablet_manager.cc:1367] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240503 14:16:03.980571 16209 raft_consensus.cc:348] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:03.980760 16211 tablet_bootstrap.cc:492] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:03.980778 16209 raft_consensus.cc:374] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.980870 16209 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.981094 16209 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:03.981557 16213 tablet_bootstrap.cc:492] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:03.981570 16209 ts_tablet_manager.cc:1398] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:16:03.981709 16213 ts_tablet_manager.cc:1367] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240503 14:16:03.981840 16209 tablet_bootstrap.cc:492] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:03.982401 16211 tablet_bootstrap.cc:654] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.982478 16213 raft_consensus.cc:348] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:03.982684 16213 raft_consensus.cc:374] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.982774 16213 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.983021 16213 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:03.983399 16209 tablet_bootstrap.cc:654] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.983605 16213 ts_tablet_manager.cc:1398] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:03.983958 16213 tablet_bootstrap.cc:492] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:03.986202 16213 tablet_bootstrap.cc:654] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.987982 16208 tablet_bootstrap.cc:492] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:03.988149 16208 ts_tablet_manager.cc:1367] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.001s
I20240503 14:16:03.988480 16211 tablet_bootstrap.cc:492] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:03.988631 16211 ts_tablet_manager.cc:1367] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 14:16:03.989054 16208 raft_consensus.cc:348] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:03.989359 16208 raft_consensus.cc:374] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.989457 16208 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.989853 16208 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:03.990022 16211 raft_consensus.cc:348] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:03.990281 16211 raft_consensus.cc:374] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.990358 16211 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.990478 16208 ts_tablet_manager.cc:1398] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:16:03.990568 16211 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:03.990994 16208 tablet_bootstrap.cc:492] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:03.991075 16211 ts_tablet_manager.cc:1398] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:16:03.991957 16149 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:03.992111 16149 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:03.992725 16209 tablet_bootstrap.cc:492] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:03.992872 16209 ts_tablet_manager.cc:1367] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:16:03.992872 16149 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:03.993041 16208 tablet_bootstrap.cc:654] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:03.993598 16209 raft_consensus.cc:348] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:03.993801 16209 raft_consensus.cc:374] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.993877 16209 raft_consensus.cc:729] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.993829 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:03.994045 14755 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:03.994092 16209 consensus_queue.cc:260] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:03.994431 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:03.994603 16209 ts_tablet_manager.cc:1398] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:03.994637 14624 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:03.995232 14819 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:03.995548 16149 raft_consensus.cc:2781] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:03.995668 16149 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:03.995759 16149 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:03.995805 16213 tablet_bootstrap.cc:492] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:03.995956 16213 ts_tablet_manager.cc:1367] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:16:03.996819 16213 raft_consensus.cc:348] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:03.997062 16213 raft_consensus.cc:374] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:03.997169 16213 raft_consensus.cc:729] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:03.997658 16213 consensus_queue.cc:260] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:03.998898 16213 ts_tablet_manager.cc:1398] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:04.001238 16149 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:04.002045 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:04.002264 14755 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.003132 16208 tablet_bootstrap.cc:492] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:04.003309 16208 ts_tablet_manager.cc:1367] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240503 14:16:04.004112 16208 raft_consensus.cc:348] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:04.004326 16208 raft_consensus.cc:374] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:04.004441 16208 raft_consensus.cc:729] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:04.004667 16208 consensus_queue.cc:260] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:04.004768 14755 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:04.005004 16149 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:04.005230 14820 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:04.005429 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:04.005499 16149 raft_consensus.cc:2781] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:04.005613 16149 raft_consensus.cc:686] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:04.005686 14624 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.005868 16149 consensus_queue.cc:237] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:04.007377 16208 ts_tablet_manager.cc:1398] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:16:04.008211 14624 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:04.016368 16135 raft_consensus.cc:482] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:04.016518 16135 raft_consensus.cc:504] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:04.016403 14481 catalog_manager.cc:5484] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:04.017071 16135 leader_election.cc:288] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:04.017539 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:16:04.017601 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:04.017776 14886 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:16:04.017840 15016 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 0.
I20240503 14:16:04.018353 14689 leader_election.cc:302] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:04.018600 16135 raft_consensus.cc:2781] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:04.018743 16135 raft_consensus.cc:482] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:04.018853 16135 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.020843 16135 raft_consensus.cc:504] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:04.021289 16135 leader_election.cc:288] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:04.021761 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:04.021781 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:04.021956 14885 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.021958 15016 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.023844 15016 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:16:04.023950 14885 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 1.
I20240503 14:16:04.024252 14689 leader_election.cc:302] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:04.024489 16135 raft_consensus.cc:2781] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:04.024672 16135 raft_consensus.cc:686] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:04.025077 16135 consensus_queue.cc:237] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:04.027508 14481 catalog_manager.cc:5484] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 0 to 1, leader changed from <none> to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 1 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:04.099409 15928 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:04.099597 15928 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:04.100131 15928 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:04.100633 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:04.100732 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:16:04.100876 14624 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:04.100966 14885 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:04.101306 14949 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:04.101563 15928 raft_consensus.cc:2781] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:04.101653 15928 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:04.101751 15928 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.103981 15928 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:04.104483 15928 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:04.104831 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:04.104897 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:04.105007 14624 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.105108 14885 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:04.107198 14624 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:04.107259 14885 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:04.107584 14949 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:04.107847 15928 raft_consensus.cc:2781] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:04.108225 15928 raft_consensus.cc:686] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:04.108512 15928 consensus_queue.cc:237] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:04.111577 14479 catalog_manager.cc:5484] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:04.454195 16135 consensus_queue.cc:1035] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:04.457617 16221 consensus_queue.cc:1035] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:04.464541 16221 consensus_queue.cc:1035] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:04.472084 16135 consensus_queue.cc:1035] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:04.499617 16225 consensus_queue.cc:1035] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:04.502022 16225 consensus_queue.cc:1035] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:04.515504 16227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:04.515988 16227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:16:04.535804 16227 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20240503 14:16:04.538481 14689 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.20.240.68:36453, user_credentials={real_user=slave}} blocked reactor thread for 64118.2us
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 293.452 ms
time per row: 0.146726 ms
I20240503 14:16:04.995409 14481 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:16:05.004226 14605 tablet_service.cc:1467] Processing CreateTablet for tablet e84474717eb440f3ad48c19a408a467b (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.004565 14994 tablet_service.cc:1467] Processing CreateTablet for tablet e84474717eb440f3ad48c19a408a467b (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.004673 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84474717eb440f3ad48c19a408a467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.004570 14736 tablet_service.cc:1467] Processing CreateTablet for tablet e84474717eb440f3ad48c19a408a467b (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.004937 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84474717eb440f3ad48c19a408a467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.004953 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e84474717eb440f3ad48c19a408a467b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.009434 16296 tablet_bootstrap.cc:492] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:05.009799 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 1e04912cfaf1494295de9f9969fc3c58 (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.010277 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e04912cfaf1494295de9f9969fc3c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.011024 16297 tablet_bootstrap.cc:492] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:05.011488 16296 tablet_bootstrap.cc:654] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.012964 16297 tablet_bootstrap.cc:654] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.015688 14734 tablet_service.cc:1467] Processing CreateTablet for tablet 1e04912cfaf1494295de9f9969fc3c58 (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.015900 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 1e04912cfaf1494295de9f9969fc3c58 (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.016084 14734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e04912cfaf1494295de9f9969fc3c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.016101 16297 tablet_bootstrap.cc:492] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:05.016248 16297 ts_tablet_manager.cc:1367] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:16:05.016249 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e04912cfaf1494295de9f9969fc3c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.017089 16297 raft_consensus.cc:348] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.017316 16297 raft_consensus.cc:374] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.017428 16297 raft_consensus.cc:729] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.017642 16297 consensus_queue.cc:260] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.018329 16297 ts_tablet_manager.cc:1398] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:16:05.021021 16300 tablet_bootstrap.cc:492] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:05.021917 16296 tablet_bootstrap.cc:492] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:05.021975 14866 tablet_service.cc:1467] Processing CreateTablet for tablet b9b1ac3afbb6471cb7172e438a1aa35e (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.022112 16296 ts_tablet_manager.cc:1367] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 14:16:05.022425 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b1ac3afbb6471cb7172e438a1aa35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.022862 16297 tablet_bootstrap.cc:492] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:05.023054 16296 raft_consensus.cc:348] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.023233 16296 raft_consensus.cc:374] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.023324 16296 raft_consensus.cc:729] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.023561 16296 consensus_queue.cc:260] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.024253 16300 tablet_bootstrap.cc:654] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.024987 16296 ts_tablet_manager.cc:1398] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:16:05.026567 16297 tablet_bootstrap.cc:654] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.028821 14994 tablet_service.cc:1467] Processing CreateTablet for tablet b9b1ac3afbb6471cb7172e438a1aa35e (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.029165 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b1ac3afbb6471cb7172e438a1aa35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.032152 16303 tablet_bootstrap.cc:492] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:05.033908 16303 tablet_bootstrap.cc:654] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.034323 14735 tablet_service.cc:1467] Processing CreateTablet for tablet b9b1ac3afbb6471cb7172e438a1aa35e (DEFAULT_TABLE table=rebalance_test_table_8 [id=6513618320e146d18dd1ffc3c1116cd6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:05.034818 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9b1ac3afbb6471cb7172e438a1aa35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:05.036253 16296 tablet_bootstrap.cc:492] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:05.038197 16296 tablet_bootstrap.cc:654] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.044164 16296 tablet_bootstrap.cc:492] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:05.044409 16296 ts_tablet_manager.cc:1367] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 14:16:05.045378 16297 tablet_bootstrap.cc:492] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:05.045267 16296 raft_consensus.cc:348] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.045475 16296 raft_consensus.cc:374] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.045586 16296 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.045681 16297 ts_tablet_manager.cc:1367] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.007s
I20240503 14:16:05.045974 16296 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.047084 16297 raft_consensus.cc:348] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.047240 16300 tablet_bootstrap.cc:492] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:05.047308 16297 raft_consensus.cc:374] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.047379 16300 ts_tablet_manager.cc:1367] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.013s
I20240503 14:16:05.047379 16297 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.047650 16297 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.047950 16296 ts_tablet_manager.cc:1398] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:16:05.048195 16297 ts_tablet_manager.cc:1398] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:16:05.048328 16300 raft_consensus.cc:348] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.048580 16300 raft_consensus.cc:374] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.048712 16300 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.048950 16300 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.049481 16300 ts_tablet_manager.cc:1398] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:16:05.049763 16300 tablet_bootstrap.cc:492] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:05.049837 16303 tablet_bootstrap.cc:492] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:05.050016 16303 ts_tablet_manager.cc:1367] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240503 14:16:05.050832 16303 raft_consensus.cc:348] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.051081 16303 raft_consensus.cc:374] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.051229 16303 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.051551 16303 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.052127 16300 tablet_bootstrap.cc:654] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.052388 16303 ts_tablet_manager.cc:1398] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:16:05.052829 16303 tablet_bootstrap.cc:492] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:05.054281 16300 tablet_bootstrap.cc:492] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:05.054556 16303 tablet_bootstrap.cc:654] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.054973 16300 ts_tablet_manager.cc:1367] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:16:05.055692 16300 raft_consensus.cc:348] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.055977 16300 raft_consensus.cc:374] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.056056 16300 raft_consensus.cc:729] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.056305 16300 consensus_queue.cc:260] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.056372 16303 tablet_bootstrap.cc:492] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:05.056536 16303 ts_tablet_manager.cc:1367] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:16:05.056821 16300 ts_tablet_manager.cc:1398] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:16:05.057202 16300 tablet_bootstrap.cc:492] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:05.057582 16303 raft_consensus.cc:348] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.057806 16303 raft_consensus.cc:374] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.057886 16303 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.058113 16303 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.058574 16303 ts_tablet_manager.cc:1398] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:16:05.058836 16300 tablet_bootstrap.cc:654] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:05.060577 16300 tablet_bootstrap.cc:492] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:05.060714 16300 ts_tablet_manager.cc:1367] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:16:05.061626 16300 raft_consensus.cc:348] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.061815 16300 raft_consensus.cc:374] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:05.061900 16300 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:05.062093 16300 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.062489 16300 ts_tablet_manager.cc:1398] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:16:05.070506 16149 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:05.070647 16149 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.071125 16149 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:05.071566 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:05.071643 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:05.071799 14755 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:05.071867 15016 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:05.072253 14820 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:05.072525 16149 raft_consensus.cc:2781] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:05.072623 16149 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:05.072716 16149 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.074605 16149 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.075183 16149 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:05.075536 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:05.075649 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:05.075773 15015 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.075810 14755 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.077524 14755 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:05.077623 15015 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:05.077885 14820 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:05.078111 16149 raft_consensus.cc:2781] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:05.078280 16149 raft_consensus.cc:686] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:05.078512 16149 consensus_queue.cc:237] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:05.080829 14484 catalog_manager.cc:5484] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:05.348879 16149 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:05.349107 16149 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.349812 16149 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:05.350275 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:05.350294 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:05.350538 14755 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:05.350553 14624 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:05.350889 14820 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:05.351225 16149 raft_consensus.cc:2781] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:05.351347 16149 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:05.351425 16149 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.353384 16149 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.353904 16149 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:05.354230 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:05.354352 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:05.354451 14755 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.354559 14624 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.356258 14755 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:05.356410 14624 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:05.356623 14820 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:05.356910 16149 raft_consensus.cc:2781] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:05.357321 16149 raft_consensus.cc:686] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:05.357584 16149 consensus_queue.cc:237] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:05.364360 14484 catalog_manager.cc:5484] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:05.431929 16245 raft_consensus.cc:482] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:05.432101 16245 raft_consensus.cc:504] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.432729 16245 leader_election.cc:288] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:05.433043 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:05.433168 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:05.433271 14755 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:16:05.433389 15015 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:16:05.433673 14559 leader_election.cc:302] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:05.433920 16245 raft_consensus.cc:2781] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:05.434051 16245 raft_consensus.cc:482] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:05.434136 16245 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.436008 16245 raft_consensus.cc:504] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.436522 16245 leader_election.cc:288] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:05.436962 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:05.436945 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:05.437137 15015 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.437523 14755 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:05.439687 15015 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:16:05.440075 14558 leader_election.cc:302] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:05.440129 14755 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:16:05.440385 16312 raft_consensus.cc:2781] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:05.440559 16312 raft_consensus.cc:686] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:05.440881 16312 consensus_queue.cc:237] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:05.443699 14484 catalog_manager.cc:5484] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:05.475595 16241 consensus_queue.cc:1035] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:05.482620 16310 consensus_queue.cc:1035] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:05.807848 16316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:05.808182 16316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:16:05.815891 16316 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240503 14:16:05.913664 16310 consensus_queue.cc:1035] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:05.929420 16310 consensus_queue.cc:1035] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:05.972777 15015 raft_consensus.cc:1261] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:16:05.973402 16312 consensus_queue.cc:1035] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:05.978909 14756 raft_consensus.cc:1261] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:16:05.979650 16312 consensus_queue.cc:1035] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 240.507 ms
time per row: 0.120253 ms
I20240503 14:16:06.207679 14483 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52568:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:16:06.215862 14605 tablet_service.cc:1467] Processing CreateTablet for tablet 67818fe8ebae4b89bfa63f9222591c0a (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.216416 14605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67818fe8ebae4b89bfa63f9222591c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.216384 14994 tablet_service.cc:1467] Processing CreateTablet for tablet 67818fe8ebae4b89bfa63f9222591c0a (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.216826 14994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67818fe8ebae4b89bfa63f9222591c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.217777 14603 tablet_service.cc:1467] Processing CreateTablet for tablet 435b95c19dfe45ceb5b15470a2ebd35e (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.218158 14604 tablet_service.cc:1467] Processing CreateTablet for tablet c8a328f25a3a4da4b4aa86b00ac015cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.218580 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 435b95c19dfe45ceb5b15470a2ebd35e (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.218165 14603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435b95c19dfe45ceb5b15470a2ebd35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.218645 14735 tablet_service.cc:1467] Processing CreateTablet for tablet c8a328f25a3a4da4b4aa86b00ac015cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.218950 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435b95c19dfe45ceb5b15470a2ebd35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.219053 14735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a328f25a3a4da4b4aa86b00ac015cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.222129 14604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a328f25a3a4da4b4aa86b00ac015cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.222609 14996 tablet_service.cc:1467] Processing CreateTablet for tablet c8a328f25a3a4da4b4aa86b00ac015cd (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.222970 14996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a328f25a3a4da4b4aa86b00ac015cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.226653 14865 tablet_service.cc:1467] Processing CreateTablet for tablet 435b95c19dfe45ceb5b15470a2ebd35e (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.227073 14865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 435b95c19dfe45ceb5b15470a2ebd35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.231047 14866 tablet_service.cc:1467] Processing CreateTablet for tablet 67818fe8ebae4b89bfa63f9222591c0a (DEFAULT_TABLE table=rebalance_test_table_9 [id=4caad855728d4b30bc82cb377d3a7a2d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:16:06.231380 14866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67818fe8ebae4b89bfa63f9222591c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:16:06.232923 16377 tablet_bootstrap.cc:492] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:06.234007 16376 tablet_bootstrap.cc:492] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:06.234871 16377 tablet_bootstrap.cc:654] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.236193 16376 tablet_bootstrap.cc:654] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.237171 16377 tablet_bootstrap.cc:492] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:06.237406 16377 ts_tablet_manager.cc:1367] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:16:06.238502 16377 raft_consensus.cc:348] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.238900 16377 raft_consensus.cc:374] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.238970 16376 tablet_bootstrap.cc:492] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:06.239080 16377 raft_consensus.cc:729] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.239148 16376 ts_tablet_manager.cc:1367] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:16:06.239588 16377 consensus_queue.cc:260] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.239965 16376 raft_consensus.cc:348] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.240188 16376 raft_consensus.cc:374] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.240226 16377 ts_tablet_manager.cc:1398] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:06.240316 16376 raft_consensus.cc:729] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.240597 16376 consensus_queue.cc:260] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.242172 16377 tablet_bootstrap.cc:492] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:06.244680 16377 tablet_bootstrap.cc:654] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.244767 16376 ts_tablet_manager.cc:1398] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:16:06.245949 16376 tablet_bootstrap.cc:492] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:06.248064 16376 tablet_bootstrap.cc:654] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.250932 16376 tablet_bootstrap.cc:492] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:06.251156 16376 ts_tablet_manager.cc:1367] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:16:06.251953 16376 raft_consensus.cc:348] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.252323 16376 raft_consensus.cc:374] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.252519 16376 raft_consensus.cc:729] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.253782 16377 tablet_bootstrap.cc:492] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:06.254000 16377 ts_tablet_manager.cc:1367] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240503 14:16:06.254145 16376 consensus_queue.cc:260] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.254961 16376 ts_tablet_manager.cc:1398] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:16:06.255115 16377 raft_consensus.cc:348] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.255375 16376 tablet_bootstrap.cc:492] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810: Bootstrap starting.
I20240503 14:16:06.255424 16377 raft_consensus.cc:374] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.255512 16377 raft_consensus.cc:729] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.255791 16377 consensus_queue.cc:260] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.258409 16377 ts_tablet_manager.cc:1398] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:16:06.258752 16377 tablet_bootstrap.cc:492] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd: Bootstrap starting.
I20240503 14:16:06.260330 16377 tablet_bootstrap.cc:654] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.267444 16376 tablet_bootstrap.cc:654] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.269832 16377 tablet_bootstrap.cc:492] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd: No bootstrap required, opened a new log
I20240503 14:16:06.270045 16377 ts_tablet_manager.cc:1367] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 14:16:06.270761 16383 tablet_bootstrap.cc:492] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194: Bootstrap starting.
I20240503 14:16:06.271028 16377 raft_consensus.cc:348] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.271314 16377 raft_consensus.cc:374] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.271543 16377 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.272647 16377 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.272845 16383 tablet_bootstrap.cc:654] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.274111 16377 ts_tablet_manager.cc:1398] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:16:06.278417 16376 tablet_bootstrap.cc:492] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810: No bootstrap required, opened a new log
I20240503 14:16:06.278587 16376 ts_tablet_manager.cc:1367] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.000s
I20240503 14:16:06.279261 16376 raft_consensus.cc:348] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.279335 16383 tablet_bootstrap.cc:492] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194: No bootstrap required, opened a new log
I20240503 14:16:06.279410 16376 raft_consensus.cc:374] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.279471 16383 ts_tablet_manager.cc:1367] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:16:06.279505 16376 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.279727 16376 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.280263 16376 ts_tablet_manager.cc:1398] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:06.281009 16382 tablet_bootstrap.cc:492] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:06.282997 16382 tablet_bootstrap.cc:654] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.284009 16383 raft_consensus.cc:348] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.284266 16383 raft_consensus.cc:374] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.284363 16383 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.284566 16383 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.285106 16383 ts_tablet_manager.cc:1398] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:16:06.286576 16382 tablet_bootstrap.cc:492] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:06.286712 16382 ts_tablet_manager.cc:1367] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:16:06.287412 16382 raft_consensus.cc:348] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.287636 16382 raft_consensus.cc:374] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.287741 16382 raft_consensus.cc:729] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.287981 16382 consensus_queue.cc:260] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.288559 16382 ts_tablet_manager.cc:1398] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:16:06.288945 16382 tablet_bootstrap.cc:492] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8: Bootstrap starting.
I20240503 14:16:06.290215 16382 tablet_bootstrap.cc:654] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8: Neither blocks nor log segments found. Creating new log.
I20240503 14:16:06.291999 16382 tablet_bootstrap.cc:492] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8: No bootstrap required, opened a new log
I20240503 14:16:06.292148 16382 ts_tablet_manager.cc:1367] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:16:06.292737 16382 raft_consensus.cc:348] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.292950 16382 raft_consensus.cc:374] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:16:06.293023 16382 raft_consensus.cc:729] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Initialized, Role: FOLLOWER
I20240503 14:16:06.293236 16382 consensus_queue.cc:260] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.293670 16382 ts_tablet_manager.cc:1398] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:16:06.346689 16357 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:06.346830 16357 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.347343 16357 leader_election.cc:288] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:06.347896 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:06.347973 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:06.348207 15015 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:06.348208 14624 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 0.
I20240503 14:16:06.348690 14819 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:06.348971 16357 raft_consensus.cc:2781] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:06.349081 16357 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:06.349179 16357 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.351068 16357 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.351563 16357 leader_election.cc:288] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:06.352051 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:06.352130 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:06.352248 14624 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.352301 15015 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.354362 15015 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:06.354580 14624 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 1.
I20240503 14:16:06.354795 14819 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:06.355111 16357 raft_consensus.cc:2781] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:06.355250 16357 raft_consensus.cc:686] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:06.355568 16357 consensus_queue.cc:237] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.358814 14484 catalog_manager.cc:5484] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 1 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:06.510887 16225 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:06.511051 16225 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.511762 16225 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:06.512146 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" is_pre_election: true
I20240503 14:16:06.512213 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810" is_pre_election: true
I20240503 14:16:06.512418 14624 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:06.512423 14756 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 0.
I20240503 14:16:06.512866 14949 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:06.513381 15928 raft_consensus.cc:2781] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:06.513489 15928 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:06.513603 15928 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.515616 16363 raft_consensus.cc:482] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:16:06.515751 16363 raft_consensus.cc:504] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.516415 16363 leader_election.cc:288] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:06.516490 15928 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.516849 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd" is_pre_election: true
I20240503 14:16:06.517093 15015 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:16:06.517143 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8" is_pre_election: true
I20240503 14:16:06.517316 15928 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:06.517338 14885 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 0.
I20240503 14:16:06.517465 14558 leader_election.cc:302] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:06.517889 16363 raft_consensus.cc:2781] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:16:06.517838 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:06.517937 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:06.518023 16363 raft_consensus.cc:482] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:16:06.518074 14756 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.518144 14624 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.518169 16363 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.520076 14756 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:06.520439 14950 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:06.520669 15928 raft_consensus.cc:2781] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:06.520828 15928 raft_consensus.cc:686] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:06.521096 14624 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 1.
I20240503 14:16:06.521068 15928 consensus_queue.cc:237] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:06.521057 16363 raft_consensus.cc:504] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.521679 16363 leader_election.cc:288] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:06.522071 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:06.522099 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:06.522281 14885 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.522286 15015 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:16:06.525113 14885 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:16:06.525102 15015 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 1.
I20240503 14:16:06.525496 14558 leader_election.cc:302] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:06.525791 16363 raft_consensus.cc:2781] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:16:06.525962 16363 raft_consensus.cc:686] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:06.526299 16363 consensus_queue.cc:237] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:06.526852 14484 catalog_manager.cc:5484] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 1 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:06.529839 14483 catalog_manager.cc:5484] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 1 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:06.738793 16357 consensus_queue.cc:1035] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:06.745875 16357 consensus_queue.cc:1035] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:16:06.870404 16392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:06.870690 16392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:16:06.877969 16392 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:16:06.898520 16225 consensus_queue.cc:1035] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:06.904157 15928 consensus_queue.cc:1035] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240503 14:16:06.931084 16363 consensus_queue.cc:1035] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:16:06.935531 16312 consensus_queue.cc:1035] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 228.835 ms
time per row: 0.114418 ms
I20240503 14:16:07.255223 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.69:35389
--local_ip_for_outbound_sockets=127.20.240.69
--tserver_master_addrs=127.20.240.126:44017
--webserver_port=43137
--webserver_interface=127.20.240.69
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240503 14:16:07.580224 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:07.580477 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:16:07.580595 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:16:07.589148 16453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:16:07.589258 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:16:07.589408 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.69
I20240503 14:16:07.598819 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.69:35389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.240.69
--webserver_port=43137
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.69
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:16:07.600127 16453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:16:07.601759 16453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:16:07.614219 16458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:07.614599 16459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:07.615629 16461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:16:07.619979 16453 server_base.cc:1034] running on GCE node
I20240503 14:16:07.620529 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:16:07.621394 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:16:07.622581 16453 hybrid_clock.cc:648] HybridClock initialized: now 1714745767622534 us; error 43 us; skew 500 ppm
I20240503 14:16:07.622876 16453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:16:07.624933 16453 webserver.cc:460] Webserver started at http://127.20.240.69:43137/ using document root <none> and password file <none>
I20240503 14:16:07.625655 16453 fs_manager.cc:359] Metadata directory not provided
I20240503 14:16:07.625782 16453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:16:07.632014 16453 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240503 14:16:07.634809 16467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:07.636147 16453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:16:07.636313 16453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e1e5858c55de4939907c3253a25fad0d"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:16:07.637241 16453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:16:07.659155 16453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:16:07.659888 16453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:16:07.660135 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:16:07.661087 16453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:16:07.662626 16453 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:16:07.662699 16453 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:07.662822 16453 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:16:07.662871 16453 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:07.694098 16453 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.69:35389
I20240503 14:16:07.694151 16579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.69:35389 every 8 connection(s)
I20240503 14:16:07.695562 16453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:16:07.695981 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 16453
I20240503 14:16:07.696736 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.70:45231
--local_ip_for_outbound_sockets=127.20.240.70
--tserver_master_addrs=127.20.240.126:44017
--webserver_port=41769
--webserver_interface=127.20.240.70
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:16:07.712110 16580 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:16:07.712376 16580 heartbeater.cc:464] Registering TS with master...
I20240503 14:16:07.713085 16580 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:16:07.714833 14476 ts_manager.cc:194] Re-registered known tserver with Master: e1e5858c55de4939907c3253a25fad0d (127.20.240.69:35389)
I20240503 14:16:07.716085 14476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.69:40015
W20240503 14:16:08.042013 16584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:08.042351 16584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:16:08.042455 16584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:16:08.050431 16584 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:16:08.050545 16584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:16:08.050786 16584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.70
I20240503 14:16:08.059787 16584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.70:45231
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.20.240.70
--webserver_port=41769
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.70
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:16:08.061295 16584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:16:08.063036 16584 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:16:08.075744 16590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:08.076305 16593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:08.077004 16591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:16:08.077772 16584 server_base.cc:1034] running on GCE node
I20240503 14:16:08.078361 16584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:16:08.079315 16584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:16:08.080529 16584 hybrid_clock.cc:648] HybridClock initialized: now 1714745768080456 us; error 64 us; skew 500 ppm
I20240503 14:16:08.080824 16584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:16:08.082995 16584 webserver.cc:460] Webserver started at http://127.20.240.70:41769/ using document root <none> and password file <none>
I20240503 14:16:08.083652 16584 fs_manager.cc:359] Metadata directory not provided
I20240503 14:16:08.083767 16584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:16:08.088567 16584 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240503 14:16:08.091135 16599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:08.092278 16584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240503 14:16:08.092401 16584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "19297c5910bf4b03b12bc4d7168475e2"
format_stamp: "Formatted at 2024-05-03 14:15:48 on dist-test-slave-xhmt"
I20240503 14:16:08.093116 16584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:16:08.118477 16584 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:16:08.119277 16584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:16:08.119585 16584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:16:08.120442 16584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:16:08.122057 16584 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:16:08.122133 16584 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:08.122277 16584 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:16:08.122339 16584 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240503 14:16:08.157058 16584 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.70:45231
I20240503 14:16:08.157109 16711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.70:45231 every 8 connection(s)
I20240503 14:16:08.158526 16584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:16:08.165362 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 16584
I20240503 14:16:08.165993 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.71:41887
--local_ip_for_outbound_sockets=127.20.240.71
--tserver_master_addrs=127.20.240.126:44017
--webserver_port=34901
--webserver_interface=127.20.240.71
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:16:08.175697 16712 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:16:08.176012 16712 heartbeater.cc:464] Registering TS with master...
I20240503 14:16:08.176923 16712 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:16:08.178313 14476 ts_manager.cc:194] Re-registered known tserver with Master: 19297c5910bf4b03b12bc4d7168475e2 (127.20.240.70:45231)
I20240503 14:16:08.179316 14476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.70:54001
W20240503 14:16:08.503445 16715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:08.503710 16715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:16:08.503842 16715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:16:08.511693 16715 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:16:08.511830 16715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:16:08.511981 16715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.71
I20240503 14:16:08.521488 16715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.71:41887
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.20.240.71
--webserver_port=34901
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.71
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:16:08.522868 16715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:16:08.524570 16715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:16:08.537971 16721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:08.538487 16720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:08.538976 16723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:16:08.539996 16715 server_base.cc:1034] running on GCE node
I20240503 14:16:08.540580 16715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:16:08.541827 16715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:16:08.544304 16715 hybrid_clock.cc:648] HybridClock initialized: now 1714745768544198 us; error 97 us; skew 500 ppm
I20240503 14:16:08.544708 16715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:16:08.548821 16715 webserver.cc:460] Webserver started at http://127.20.240.71:34901/ using document root <none> and password file <none>
I20240503 14:16:08.549782 16715 fs_manager.cc:359] Metadata directory not provided
I20240503 14:16:08.549969 16715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:16:08.555198 16715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:16:08.557905 16729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:08.559494 16715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240503 14:16:08.559638 16715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "d8ecf8e658bc4102b272f90b8146cb99"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:16:08.560396 16715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:16:08.581595 16715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:16:08.582322 16715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:16:08.582562 16715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:16:08.583397 16715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:16:08.584784 16715 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:16:08.584874 16715 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:08.584997 16715 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:16:08.585048 16715 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:08.622845 16715 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.71:41887
I20240503 14:16:08.622891 16841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.71:41887 every 8 connection(s)
I20240503 14:16:08.624295 16715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:16:08.628809 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 16715
I20240503 14:16:08.629549 21441 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
/tmp/dist-test-taskUpGD8F/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.240.72:43695
--local_ip_for_outbound_sockets=127.20.240.72
--tserver_master_addrs=127.20.240.126:44017
--webserver_port=36977
--webserver_interface=127.20.240.72
--builtin_ntp_servers=127.20.240.84:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240503 14:16:08.638331 16842 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:16:08.638624 16842 heartbeater.cc:464] Registering TS with master...
I20240503 14:16:08.639317 16842 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:16:08.640648 14476 ts_manager.cc:194] Re-registered known tserver with Master: d8ecf8e658bc4102b272f90b8146cb99 (127.20.240.71:41887)
I20240503 14:16:08.641800 14476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.71:45841
I20240503 14:16:08.718657 16580 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
W20240503 14:16:08.981083 16845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:16:08.981391 16845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:16:08.981523 16845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:16:08.988766 16845 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240503 14:16:08.988906 16845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:16:08.989087 16845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.240.72
I20240503 14:16:08.998555 16845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.240.84:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.240.72:43695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.20.240.72
--webserver_port=36977
--tserver_master_addrs=127.20.240.126:44017
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.240.72
--log_dir=/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:16:08.999846 16845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:16:09.001457 16845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:16:09.013171 16852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:09.013878 16853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:16:09.014647 16855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:16:09.015799 16845 server_base.cc:1034] running on GCE node
I20240503 14:16:09.016450 16845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:16:09.017463 16845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:16:09.018669 16845 hybrid_clock.cc:648] HybridClock initialized: now 1714745769018631 us; error 37 us; skew 500 ppm
I20240503 14:16:09.018987 16845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:16:09.021251 16845 webserver.cc:460] Webserver started at http://127.20.240.72:36977/ using document root <none> and password file <none>
I20240503 14:16:09.021956 16845 fs_manager.cc:359] Metadata directory not provided
I20240503 14:16:09.022070 16845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:16:09.027256 16845 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:16:09.030644 16861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:09.032054 16845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240503 14:16:09.032225 16845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ebe953d95ec24ec7a9071fb834984cce"
format_stamp: "Formatted at 2024-05-03 14:15:49 on dist-test-slave-xhmt"
I20240503 14:16:09.032963 16845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:16:09.065368 16845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:16:09.066215 16845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:16:09.066521 16845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:16:09.067468 16845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:16:09.068992 16845 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:16:09.069069 16845 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:09.069166 16845 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:16:09.069213 16845 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:16:09.100831 16845 rpc_server.cc:307] RPC server started. Bound to: 127.20.240.72:43695
I20240503 14:16:09.100854 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.240.72:43695 every 8 connection(s)
I20240503 14:16:09.102262 16845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUpGD8F/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714745515740426-21441-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:16:09.110186 21441 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUpGD8F/build/asan/bin/kudu as pid 16845
I20240503 14:16:09.116477 16974 heartbeater.cc:347] Connected to a master server at 127.20.240.126:44017
I20240503 14:16:09.116779 16974 heartbeater.cc:464] Registering TS with master...
I20240503 14:16:09.117645 16974 heartbeater.cc:511] Master 127.20.240.126:44017 requested a full tablet report, sending...
I20240503 14:16:09.119148 14483 ts_manager.cc:194] Re-registered known tserver with Master: ebe953d95ec24ec7a9071fb834984cce (127.20.240.72:43695)
I20240503 14:16:09.120338 14483 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.240.72:39663
I20240503 14:16:09.181499 16712 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
I20240503 14:16:09.181767 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.182018 14624 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.182168 14624 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.185038 14624 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.186053 14624 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.186412 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.186645 14756 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.186821 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.187021 15015 raft_consensus.cc:3032] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.187116 15015 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.187428 15015 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.187932 15015 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.188921 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.189128 14625 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.189224 14625 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.189472 14756 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.189927 14559 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.190412 16312 raft_consensus.cc:2781] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.190598 15015 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.191576 16312 raft_consensus.cc:686] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.191823 14625 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.191874 16312 consensus_queue.cc:237] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.192557 14625 leader_election.cc:288] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.193025 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.193222 14884 raft_consensus.cc:3032] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.193193 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.193303 14884 raft_consensus.cc:729] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.193393 15015 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.193593 14884 consensus_queue.cc:260] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.193816 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.194000 14625 raft_consensus.cc:465] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.194141 14884 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.194810 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.194991 14624 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.195097 14624 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.196130 15015 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.196530 14558 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.196763 14884 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.197737 14624 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.198607 16312 raft_consensus.cc:2781] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.198894 16312 raft_consensus.cc:686] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.198987 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.199143 14884 raft_consensus.cc:3032] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.199235 14884 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.199208 16312 consensus_queue.cc:237] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.199467 14884 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.199817 14884 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.200577 14483 catalog_manager.cc:5484] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.202600 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.202831 14756 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.205387 14884 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.205976 14558 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.206300 16985 raft_consensus.cc:2781] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.207486 14756 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.208071 14624 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.208380 16985 raft_consensus.cc:686] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.208755 16985 consensus_queue.cc:237] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.209484 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46d6b19383c449198147cf7a999199d1"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.209728 14625 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.209942 14625 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.212862 14625 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.214287 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.214479 14884 raft_consensus.cc:3032] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.214577 14884 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.214531 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.214792 14756 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.214838 14884 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.215160 14884 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.216931 14884 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.217342 14558 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.217379 14756 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.217694 16312 raft_consensus.cc:2781] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.217950 16312 raft_consensus.cc:686] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.217957 14625 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.219151 16312 consensus_queue.cc:237] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.220052 14623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "843132c2c55645d598035edd0d2233c3"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.220254 14623 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.220360 14623 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.223733 14623 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.224354 14623 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.224916 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.225090 14884 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.225195 14623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.225234 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.225354 14623 raft_consensus.cc:482] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.225458 14623 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.225459 15015 raft_consensus.cc:3032] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.225545 15015 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.225806 15015 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.226173 15015 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.227681 14884 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.227998 14623 raft_consensus.cc:504] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.228986 14623 leader_election.cc:288] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.229250 15015 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.230141 14558 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.230365 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.230522 14624 raft_consensus.cc:482] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.230630 14624 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.231194 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.231456 15016 raft_consensus.cc:3032] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.231567 15016 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.231848 15016 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.232306 15016 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.233282 14624 raft_consensus.cc:504] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.233882 14624 leader_election.cc:288] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.234707 14623 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.234879 14623 raft_consensus.cc:465] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.235131 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.235401 14884 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.235697 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.235885 14622 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.235989 14622 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.238639 14622 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.238909 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.239107 15014 raft_consensus.cc:3032] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.239197 15014 raft_consensus.cc:729] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.239421 15014 consensus_queue.cc:260] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.239746 15014 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.239763 14622 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.240578 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.240741 14622 raft_consensus.cc:465] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.241498 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.241669 14622 raft_consensus.cc:465] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.242542 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.242601 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.242758 14755 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.242758 14622 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.242859 14622 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.243731 15016 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.244470 14884 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.244894 14558 leader_election.cc:302] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.245419 14755 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.245745 16989 raft_consensus.cc:2781] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.245852 14559 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.246608 14558 leader_election.cc:302] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.247423 16985 raft_consensus.cc:2781] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.242313 15014 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.252272 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.253023 15016 raft_consensus.cc:3032] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.253386 15016 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.253923 15016 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.255470 15016 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.245448 14622 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.257364 14622 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.260522 15016 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.260587 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.260843 14756 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.263484 14756 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.263722 16997 raft_consensus.cc:2781] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.263881 16997 raft_consensus.cc:686] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.264326 16997 consensus_queue.cc:237] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.265262 16985 raft_consensus.cc:686] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.265514 16985 consensus_queue.cc:237] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.268764 16312 raft_consensus.cc:2781] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.268956 16312 raft_consensus.cc:686] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.269287 16312 consensus_queue.cc:237] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.270176 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.270385 14755 raft_consensus.cc:3032] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.270490 14755 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.270882 14755 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.271333 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.271529 14624 raft_consensus.cc:482] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.271644 14624 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.271653 14755 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.274509 14755 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.274865 16989 raft_consensus.cc:686] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.275015 14559 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.274662 14624 raft_consensus.cc:504] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.275404 16985 raft_consensus.cc:2781] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.275677 16985 raft_consensus.cc:686] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.276041 16985 consensus_queue.cc:237] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.276259 14624 leader_election.cc:288] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.277179 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.277168 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.277369 14625 raft_consensus.cc:465] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.277401 14755 raft_consensus.cc:3032] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.277570 14755 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.277832 14755 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.278185 14755 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.278177 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.278362 14625 raft_consensus.cc:465] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.278486 16989 consensus_queue.cc:237] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.279050 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.280992 14625 raft_consensus.cc:482] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.281100 14625 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.283793 14625 raft_consensus.cc:504] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.284700 14625 leader_election.cc:288] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:09.285631 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.285852 14625 raft_consensus.cc:465] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.286914 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.287163 14625 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.287353 14625 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.287590 14755 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.288264 14559 leader_election.cc:302] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.288846 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.289067 14884 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.289888 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.290125 15016 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.290400 16989 raft_consensus.cc:2781] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.290585 16989 raft_consensus.cc:686] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.290827 16989 consensus_queue.cc:237] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.291843 14884 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.292383 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.292577 15014 raft_consensus.cc:3032] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.292665 15014 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.292927 15014 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.293270 15014 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.294272 14625 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.295833 15014 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.295850 14625 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:09.297881 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.298089 14884 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.299104 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.299264 14624 raft_consensus.cc:482] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.299405 14624 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.300598 14884 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.302376 14624 raft_consensus.cc:504] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.302624 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.302973 14624 leader_election.cc:288] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.303171 14755 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.304879 14755 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.305307 14558 leader_election.cc:302] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.305271 14559 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.305655 16989 raft_consensus.cc:2781] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.305505 14483 catalog_manager.cc:5484] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.305934 15016 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.306063 14483 catalog_manager.cc:5484] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.306345 16997 raft_consensus.cc:2781] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.306505 16997 raft_consensus.cc:686] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.306725 16997 consensus_queue.cc:237] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.306931 14483 catalog_manager.cc:5484] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.307364 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.307379 14483 catalog_manager.cc:5484] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.307619 14624 raft_consensus.cc:482] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.307871 14624 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.307829 14483 catalog_manager.cc:5484] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.308276 14483 catalog_manager.cc:5484] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.308766 14483 catalog_manager.cc:5484] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.309168 14483 catalog_manager.cc:5484] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.309587 14483 catalog_manager.cc:5484] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.311764 14624 raft_consensus.cc:504] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.311942 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.312119 14884 raft_consensus.cc:3032] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.312224 14884 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.312466 14884 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.312848 14884 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.313021 14624 leader_election.cc:288] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.314577 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.314796 14624 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.314952 14624 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.315505 14884 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.315898 16989 raft_consensus.cc:686] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.316625 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.316812 14884 raft_consensus.cc:3032] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.316907 14884 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.317175 14884 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.317504 14884 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.317971 14624 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.318617 14624 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.319196 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.319419 14756 raft_consensus.cc:3032] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.319543 14756 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.319705 14884 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.319816 14756 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.320147 16989 consensus_queue.cc:237] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.320277 14756 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.321913 14558 leader_election.cc:302] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.322153 16997 raft_consensus.cc:2781] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.322314 16997 raft_consensus.cc:686] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.322561 16997 consensus_queue.cc:237] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.322924 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.323148 15014 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.325029 14756 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.325304 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:09.325572 14624 raft_consensus.cc:482] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.325742 14624 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.326581 15014 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.326581 14559 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.326885 16996 raft_consensus.cc:2781] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.327044 16996 raft_consensus.cc:686] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.327065 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.327239 14884 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.327533 16996 consensus_queue.cc:237] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.322772 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.327877 15016 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.329484 14624 raft_consensus.cc:504] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.330251 14624 leader_election.cc:288] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.330711 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.330880 14756 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.331805 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.331977 15014 raft_consensus.cc:3032] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.332048 15014 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.332319 15014 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.332648 15014 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.334014 14756 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.334401 14559 leader_election.cc:302] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.334648 16985 raft_consensus.cc:2781] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.334775 16985 raft_consensus.cc:686] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.334985 16985 consensus_queue.cc:237] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.335256 15014 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.335691 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.335902 14755 raft_consensus.cc:3032] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.335999 14755 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.336359 14755 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.336863 14755 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.338802 14558 leader_election.cc:302] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.339147 16312 raft_consensus.cc:2781] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.339304 16312 raft_consensus.cc:686] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.339607 16312 consensus_queue.cc:237] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.345564 14483 catalog_manager.cc:5484] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.346333 14483 catalog_manager.cc:5484] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:09.346964 14884 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.345743 15016 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.350253 14755 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 2.
I20240503 14:16:09.354660 14483 catalog_manager.cc:5484] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.355130 14483 catalog_manager.cc:5484] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.355473 14483 catalog_manager.cc:5484] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.355737 14483 catalog_manager.cc:5484] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 2 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.388530 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.388758 14755 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.388857 14755 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.390959 14755 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.391561 14755 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.392436 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.392587 14755 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.392706 14755 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.392608 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.392870 15016 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.394435 14755 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.394971 14755 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.395859 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.396039 14755 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.396131 14755 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.396332 15016 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.396852 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.397099 14624 raft_consensus.cc:3032] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.397245 14624 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.397588 14624 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.398104 14624 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.398536 14755 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.399212 14755 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.399976 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46d6b19383c449198147cf7a999199d1"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.400116 14755 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.400239 14755 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.401101 14624 raft_consensus.cc:2396] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.401877 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.402125 14884 raft_consensus.cc:3032] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.402029 14755 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.402271 14884 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.402570 14884 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.402705 14755 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.402740 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.402930 15014 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.402930 14884 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.404623 14884 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.402266 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.404899 14885 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.405550 15014 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.406663 14885 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.406672 14689 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.407063 14689 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:09.407300 16585 raft_consensus.cc:2781] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.407300 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.407629 14755 raft_consensus.cc:482] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.407816 14755 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.408358 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.408587 14625 raft_consensus.cc:3032] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.408550 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.408674 14625 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.408712 14885 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.408931 14625 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.409238 14625 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.410673 14755 raft_consensus.cc:504] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.411248 14885 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.411284 14755 leader_election.cc:288] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.412104 14625 raft_consensus.cc:2396] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.412261 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.412513 14755 raft_consensus.cc:482] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.412715 14755 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.413120 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.413332 14624 raft_consensus.cc:3032] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.413435 14624 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.413697 14624 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.413990 14624 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.415892 14755 raft_consensus.cc:504] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.416733 14755 leader_election.cc:288] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.417677 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.417973 14755 raft_consensus.cc:482] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.418118 14755 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.420729 14624 raft_consensus.cc:2396] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.421003 14755 raft_consensus.cc:504] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.421864 14755 leader_election.cc:288] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.423019 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.423297 14755 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.423509 14755 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.425210 17015 raft_consensus.cc:2781] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.425590 16585 raft_consensus.cc:686] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.425910 16585 consensus_queue.cc:237] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.426846 14755 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.427618 14755 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:09.428706 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.429021 14755 raft_consensus.cc:482] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.429209 14755 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.432358 17015 raft_consensus.cc:686] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.432543 14755 raft_consensus.cc:504] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.432703 17015 consensus_queue.cc:237] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.433261 14755 leader_election.cc:288] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.434298 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.434597 14755 raft_consensus.cc:465] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.435645 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0184718c9fac4009b834726d661642f3"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.435914 14755 raft_consensus.cc:465] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.437464 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.437651 14624 raft_consensus.cc:3032] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.437561 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.437754 14624 raft_consensus.cc:729] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.437835 15014 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.438006 14624 consensus_queue.cc:260] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.438429 14624 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.440714 15014 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.441643 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.441773 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.441980 15014 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.442016 14885 raft_consensus.cc:3032] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.442116 14885 raft_consensus.cc:729] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.442590 14885 consensus_queue.cc:260] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.443027 14885 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.446053 15014 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.446806 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.447000 14884 raft_consensus.cc:3032] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.447081 14884 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.447321 14884 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.447707 14884 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.447348 14624 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.448313 14885 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.450697 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.450913 14624 raft_consensus.cc:3032] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.451040 14624 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.451418 14884 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.451431 14624 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.451779 14624 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.451829 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.452049 15014 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.454962 15014 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.455488 14624 raft_consensus.cc:2396] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.456300 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.456513 14884 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.456797 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.457322 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.457557 14625 raft_consensus.cc:3032] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.457646 14625 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.457989 14625 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.458360 14625 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.459302 14884 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.461946 14625 raft_consensus.cc:2396] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.462683 14755 raft_consensus.cc:482] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.462713 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.462987 15014 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.462934 14689 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.463454 14755 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.464357 17015 raft_consensus.cc:2781] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.464658 17015 raft_consensus.cc:686] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.465152 17015 consensus_queue.cc:237] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.465768 14483 catalog_manager.cc:5484] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.466403 14755 raft_consensus.cc:504] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.466357 14483 catalog_manager.cc:5484] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.467113 14755 leader_election.cc:288] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.463465 14689 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.468003 14689 leader_election.cc:302] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.468310 14689 leader_election.cc:302] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.468526 14756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.468640 14689 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.468724 14756 raft_consensus.cc:482] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.468822 14756 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.469293 17014 raft_consensus.cc:2781] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.469470 17014 raft_consensus.cc:686] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.469760 17014 consensus_queue.cc:237] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.470364 15014 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.471177 16585 raft_consensus.cc:2781] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.472072 14756 raft_consensus.cc:504] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.472528 17016 raft_consensus.cc:2781] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.472667 17016 raft_consensus.cc:686] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.472772 14756 leader_election.cc:288] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.472940 17016 consensus_queue.cc:237] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.473759 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.473963 14755 raft_consensus.cc:465] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240503 14:16:09.474215 16585 raft_consensus.cc:686] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.474483 16585 consensus_queue.cc:237] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.474977 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.475185 14755 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.475288 14755 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.467978 17018 raft_consensus.cc:2781] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.476171 17018 raft_consensus.cc:686] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.476466 17018 consensus_queue.cc:237] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.478502 14755 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.479249 14755 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.481496 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.481669 14755 raft_consensus.cc:482] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.481797 14755 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.482574 14483 catalog_manager.cc:5484] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.483000 14689 leader_election.cc:302] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.483112 14483 catalog_manager.cc:5484] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.483498 14483 catalog_manager.cc:5484] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.483991 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.484170 15014 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.484864 14755 raft_consensus.cc:504] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.485502 14755 leader_election.cc:288] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.486500 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.486603 15014 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.486711 14755 raft_consensus.cc:482] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.486820 14755 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.488370 14689 leader_election.cc:302] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:09.488317 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.488605 14625 raft_consensus.cc:3032] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.488691 14689 leader_election.cc:302] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.488723 14625 raft_consensus.cc:729] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.489068 14625 consensus_queue.cc:260] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.489324 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.489485 14625 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.489524 14884 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.490396 16585 raft_consensus.cc:2781] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.490512 16585 raft_consensus.cc:686] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.490739 16585 consensus_queue.cc:237] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.492239 14884 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.492411 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.492595 15016 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.492671 17016 raft_consensus.cc:2781] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.492697 17018 raft_consensus.cc:2781] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.492851 17016 raft_consensus.cc:686] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.493024 17018 raft_consensus.cc:686] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.493098 17016 consensus_queue.cc:237] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.493240 17018 consensus_queue.cc:237] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.494170 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.494433 15014 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.495522 15016 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.495666 14689 leader_election.cc:302] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.495858 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.496003 17014 raft_consensus.cc:2781] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.496079 14624 raft_consensus.cc:3032] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.496142 17014 raft_consensus.cc:686] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.496181 14624 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.496347 17014 consensus_queue.cc:237] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.496456 14624 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.496757 14624 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.497092 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.497305 14623 raft_consensus.cc:3032] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.497387 14623 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.497668 14623 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.497967 14623 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.498831 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.499017 14622 raft_consensus.cc:3032] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.499118 14622 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.499390 14622 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.499804 14622 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.500351 14625 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.503527 14624 raft_consensus.cc:2396] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.506798 14689 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.507300 17015 raft_consensus.cc:2781] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.507450 17015 raft_consensus.cc:686] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.507761 17015 consensus_queue.cc:237] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.508308 15014 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.508774 14622 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.509171 14623 raft_consensus.cc:2396] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.512089 14755 raft_consensus.cc:504] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.512766 14755 leader_election.cc:288] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.513836 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.514017 14755 raft_consensus.cc:482] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.514156 14755 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.514830 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.515019 15014 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.517028 14755 raft_consensus.cc:504] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.517681 14755 leader_election.cc:288] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.517915 15014 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.518519 14689 leader_election.cc:302] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:09.518697 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.518896 14625 raft_consensus.cc:3032] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.518983 14625 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.519284 14625 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.519555 14625 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.519831 17014 raft_consensus.cc:2781] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.519980 17014 raft_consensus.cc:686] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.520264 17014 consensus_queue.cc:237] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.522089 14689 leader_election.cc:302] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.522415 14625 raft_consensus.cc:2396] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.523123 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.523188 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.523348 14755 raft_consensus.cc:482] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.523438 14884 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.523468 14755 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.526378 14884 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.526180 14483 catalog_manager.cc:5484] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.527297 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.527149 14483 catalog_manager.cc:5484] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.527493 14625 raft_consensus.cc:3032] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.527581 14625 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.527850 14625 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.527808 14483 catalog_manager.cc:5484] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.528193 14625 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.527148 14755 raft_consensus.cc:504] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.528430 14483 catalog_manager.cc:5484] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.528712 14755 leader_election.cc:288] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:09.529002 14483 catalog_manager.cc:5484] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.529891 14756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.530081 14756 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.530122 16585 raft_consensus.cc:2781] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.530261 14756 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.530850 14625 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.532042 14689 leader_election.cc:302] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.532285 17014 raft_consensus.cc:2781] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.532414 17014 raft_consensus.cc:686] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.532646 17014 consensus_queue.cc:237] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.532965 14756 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.533946 14756 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.534102 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.534340 15014 raft_consensus.cc:3032] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.534410 15014 raft_consensus.cc:729] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.534488 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.534680 14884 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.534703 15014 consensus_queue.cc:260] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.535048 16585 raft_consensus.cc:686] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.535099 15014 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.535459 16585 consensus_queue.cc:237] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.537211 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.537384 14755 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.537391 14884 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.537492 14755 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.538177 14689 leader_election.cc:302] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.538777 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.538985 14884 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.539142 17018 raft_consensus.cc:2781] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.539333 17018 raft_consensus.cc:686] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.539664 17018 consensus_queue.cc:237] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.540565 14755 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.541980 14884 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.542165 15014 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 2.
I20240503 14:16:09.542791 14755 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.542887 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.543098 14625 raft_consensus.cc:3032] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.543190 14625 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.543452 14625 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.543794 14625 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.543840 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:09.544103 14755 raft_consensus.cc:482] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.544234 14755 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.547060 14755 raft_consensus.cc:504] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.547243 14625 raft_consensus.cc:2396] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.547652 14755 leader_election.cc:288] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.548046 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.548255 14884 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.551039 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.551234 14625 raft_consensus.cc:3032] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.551307 14625 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.551620 14625 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.551915 14625 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.554700 14689 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:09.555322 17018 raft_consensus.cc:2781] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.555428 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.555661 15014 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.555529 17018 raft_consensus.cc:686] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.556005 17018 consensus_queue.cc:237] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.557788 14483 catalog_manager.cc:5484] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:09.558463 15014 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.558924 14689 leader_election.cc:302] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.559213 17014 raft_consensus.cc:2781] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.559409 17014 raft_consensus.cc:686] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.559427 14884 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 3.
I20240503 14:16:09.559839 14689 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.560119 17023 raft_consensus.cc:2781] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.560250 17023 raft_consensus.cc:686] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.560495 17023 consensus_queue.cc:237] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.559813 17014 consensus_queue.cc:237] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.563135 14483 catalog_manager.cc:5484] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.563928 14483 catalog_manager.cc:5484] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.564666 14483 catalog_manager.cc:5484] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.565398 14483 catalog_manager.cc:5484] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.567485 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.567683 14624 raft_consensus.cc:3032] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.567792 14624 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.568076 14624 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.568432 14624 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.570618 14625 raft_consensus.cc:2396] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.571435 14624 raft_consensus.cc:2396] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.571411 14483 catalog_manager.cc:5484] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 1 to 2, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 2 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.589397 14483 catalog_manager.cc:5484] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.590055 14483 catalog_manager.cc:5484] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.590881 14483 catalog_manager.cc:5484] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 3 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.593255 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.593497 14884 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.593623 14884 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.596307 14884 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.596997 14884 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.597582 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.597792 14755 raft_consensus.cc:3032] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.597883 14755 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.598410 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.598506 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.598709 15014 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.598866 14884 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.599058 14884 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.601424 15014 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.601984 14819 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:09.601990 14884 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.602434 16847 raft_consensus.cc:2781] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.602562 16847 raft_consensus.cc:686] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.602844 16847 consensus_queue.cc:237] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.603304 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.603513 14756 raft_consensus.cc:3032] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.603585 14756 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.603840 14756 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.604090 14756 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.598129 14755 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.604310 14755 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.607189 14755 raft_consensus.cc:2396] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.607306 14756 raft_consensus.cc:2396] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.608011 14884 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.608471 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.608675 14624 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.611641 14624 raft_consensus.cc:2396] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.612099 14819 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:09.612412 16310 raft_consensus.cc:2726] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:09.612939 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46d6b19383c449198147cf7a999199d1"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.613199 14884 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.613359 14884 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.616286 14884 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.617087 14884 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.617923 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.618119 14624 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.618180 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.618562 14755 raft_consensus.cc:3032] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.618654 14755 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.618894 14755 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.619170 14755 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.621137 14624 raft_consensus.cc:2396] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.626513 14755 raft_consensus.cc:2396] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.627103 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.627373 14884 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.627398 14820 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:09.627558 14884 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.627871 16310 raft_consensus.cc:2726] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:09.632011 14884 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.632586 14885 raft_consensus.cc:1226] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Rejecting Update request from peer e9d4c39af431443189e8475561596810 for earlier term 2. Current term is 3. Ops: []
I20240503 14:16:09.633538 16996 consensus_queue.cc:1046] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.633618 14483 catalog_manager.cc:5484] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.633867 16996 raft_consensus.cc:3032] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.633947 16996 raft_consensus.cc:729] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.634214 16996 consensus_queue.cc:260] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.634572 16996 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.637776 14884 leader_election.cc:288] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.638231 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.638528 14624 raft_consensus.cc:2396] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.638813 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.639051 15014 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.641003 15014 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.641093 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.641304 14884 raft_consensus.cc:482] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.641425 14884 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.641481 14819 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:09.641832 16847 raft_consensus.cc:2781] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.642082 16847 raft_consensus.cc:686] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.642496 16847 consensus_queue.cc:237] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.643976 14884 raft_consensus.cc:504] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.644479 16842 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
I20240503 14:16:09.644825 14884 leader_election.cc:288] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.645774 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.645942 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.646004 14755 raft_consensus.cc:3032] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.646099 14755 raft_consensus.cc:729] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.646111 14884 raft_consensus.cc:482] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.646222 14884 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.646654 14483 catalog_manager.cc:5484] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.647614 14755 consensus_queue.cc:260] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.647953 14755 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.649207 14884 raft_consensus.cc:504] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.649808 14884 leader_election.cc:288] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.650792 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.650825 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.650974 14885 raft_consensus.cc:482] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.650995 14624 raft_consensus.cc:3032] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.651089 14885 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.651089 14624 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.651391 14624 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.651690 14624 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.653573 14885 raft_consensus.cc:504] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.654599 14624 raft_consensus.cc:2396] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.655064 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.655022 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.655316 15016 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.655313 15015 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.655800 14885 leader_election.cc:288] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.656607 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.656817 14756 raft_consensus.cc:3032] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.656893 14756 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.657114 14756 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.657313 14756 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.659185 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.659396 15014 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.659878 14756 raft_consensus.cc:2396] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.661249 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.661487 14885 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.661582 15015 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.661661 14885 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.662050 14819 leader_election.cc:302] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:09.662212 14755 raft_consensus.cc:2396] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.662541 16847 raft_consensus.cc:2781] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.662678 16847 raft_consensus.cc:686] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.662966 16847 consensus_queue.cc:237] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.663812 15016 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.664011 15014 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.664883 14819 leader_election.cc:302] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:09.665210 16847 raft_consensus.cc:2781] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.665359 14819 leader_election.cc:302] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:09.665409 16847 raft_consensus.cc:686] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.665834 16310 raft_consensus.cc:2781] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.665731 16847 consensus_queue.cc:237] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.666038 16310 raft_consensus.cc:686] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.666435 16310 consensus_queue.cc:237] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.669605 14483 catalog_manager.cc:5484] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.669862 14885 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.670661 14885 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.671317 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.671554 14755 raft_consensus.cc:3032] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.671664 14755 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.671974 14755 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.672356 14755 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.676916 14755 raft_consensus.cc:2396] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.685631 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.686229 14885 raft_consensus.cc:482] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.686703 14885 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.691318 14885 raft_consensus.cc:504] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.692808 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.693054 14755 raft_consensus.cc:3032] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.693168 14755 raft_consensus.cc:729] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.693473 14755 consensus_queue.cc:260] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.693946 14755 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.696413 14885 leader_election.cc:288] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.697396 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.697645 15014 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.697696 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.697932 14624 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.698221 14483 catalog_manager.cc:5484] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:09.698812 14755 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.700372 15014 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.700673 14624 raft_consensus.cc:2396] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.701062 14819 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:09.701464 14819 leader_election.cc:302] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:09.701848 16847 raft_consensus.cc:2726] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:09.701984 16847 raft_consensus.cc:2781] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.702109 16847 raft_consensus.cc:686] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.702420 16847 consensus_queue.cc:237] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.703188 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0184718c9fac4009b834726d661642f3"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.703416 14885 raft_consensus.cc:482] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.703560 14885 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.706265 14885 raft_consensus.cc:504] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.707329 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.707525 14755 raft_consensus.cc:3032] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.707628 14755 raft_consensus.cc:729] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.707890 14755 consensus_queue.cc:260] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.708261 14755 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.709170 14483 catalog_manager.cc:5484] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.709611 14483 catalog_manager.cc:5484] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 1 to 2, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 2 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.710623 14755 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.710726 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.710934 15014 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.711045 14820 leader_election.cc:302] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.711401 14885 leader_election.cc:288] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.711402 16847 raft_consensus.cc:2781] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.711788 16847 raft_consensus.cc:686] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.711833 14884 raft_consensus.cc:1261] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.712519 16997 consensus_queue.cc:1035] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.712702 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.712893 14884 raft_consensus.cc:482] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.713023 14884 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.712731 16847 consensus_queue.cc:237] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.714360 15014 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.716436 14884 raft_consensus.cc:504] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.717314 14884 leader_election.cc:288] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.718441 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.718680 14884 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.718830 14884 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.722719 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.722931 14624 raft_consensus.cc:3032] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.723009 14624 raft_consensus.cc:729] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.723265 14624 consensus_queue.cc:260] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.723542 14884 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.723646 14624 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.724148 14884 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.725019 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.725261 14755 raft_consensus.cc:3032] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.725378 14755 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.725670 14755 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.725821 14624 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.726033 14755 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.726759 14819 leader_election.cc:302] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.727149 16847 raft_consensus.cc:2781] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.727115 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.727285 16847 raft_consensus.cc:686] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.727547 16847 consensus_queue.cc:237] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.728633 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.728847 14624 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.730931 14755 raft_consensus.cc:2396] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.730990 14624 raft_consensus.cc:2396] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.731614 14820 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:09.732132 16847 raft_consensus.cc:2726] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:09.737571 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.737972 14886 raft_consensus.cc:482] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.738122 14886 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.738896 15014 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.740211 14483 catalog_manager.cc:5484] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 1 to 2, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 2 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.740821 14886 raft_consensus.cc:504] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.741659 14886 leader_election.cc:288] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.741926 15014 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.742627 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.742868 14886 raft_consensus.cc:482] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.743005 14886 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.745538 14886 raft_consensus.cc:504] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.746107 14886 leader_election.cc:288] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.742708 14885 raft_consensus.cc:1226] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Rejecting Update request from peer e9d4c39af431443189e8475561596810 for earlier term 2. Current term is 3. Ops: []
I20240503 14:16:09.746960 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.747169 14886 raft_consensus.cc:482] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.747323 14886 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.747294 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.747599 14625 raft_consensus.cc:3032] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.747715 14625 raft_consensus.cc:729] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.748070 14625 consensus_queue.cc:260] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.748525 14625 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.748754 16996 consensus_queue.cc:1046] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.749202 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.749428 15016 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.750062 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.750810 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.751009 15014 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.751936 14625 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.752244 15016 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.753763 15014 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.753782 14886 raft_consensus.cc:504] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.755125 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.755173 14819 leader_election.cc:302] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:09.755349 14755 raft_consensus.cc:3032] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.755440 14755 raft_consensus.cc:729] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.755592 14819 leader_election.cc:302] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:09.755693 14755 consensus_queue.cc:260] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.755889 17053 raft_consensus.cc:2781] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.756016 17053 raft_consensus.cc:686] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.756039 14755 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.756254 16847 raft_consensus.cc:2781] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.756312 17053 consensus_queue.cc:237] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.758760 14755 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.759310 14820 leader_election.cc:302] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:09.759651 16310 raft_consensus.cc:2781] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.759766 16310 raft_consensus.cc:686] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.759987 16310 consensus_queue.cc:237] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.760926 14483 catalog_manager.cc:5484] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 1 to 2, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 2 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.763649 14886 leader_election.cc:288] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.764729 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.765090 14886 raft_consensus.cc:482] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.765385 14886 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.768345 16847 raft_consensus.cc:686] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.768652 16847 consensus_queue.cc:237] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.771361 14886 raft_consensus.cc:504] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.772078 15015 raft_consensus.cc:1226] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Rejecting Update request from peer e9d4c39af431443189e8475561596810 for earlier term 2. Current term is 3. Ops: []
I20240503 14:16:09.772296 14476 catalog_manager.cc:5484] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 1 to 2, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 2 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.772614 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.772825 15015 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.772766 16985 consensus_queue.cc:1046] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.772850 14476 catalog_manager.cc:5484] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:09.773092 14624 raft_consensus.cc:3032] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.773208 14624 raft_consensus.cc:729] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.773474 14624 consensus_queue.cc:260] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.773831 14624 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.774142 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.774463 15014 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.775591 15015 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.775924 15016 raft_consensus.cc:1226] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Rejecting Update request from peer e9d4c39af431443189e8475561596810 for earlier term 1. Current term is 2. Ops: []
I20240503 14:16:09.776593 14819 leader_election.cc:302] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:09.776753 16312 consensus_queue.cc:1046] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.776912 16847 raft_consensus.cc:2781] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:16:09.777047 16847 raft_consensus.cc:686] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.777041 16996 raft_consensus.cc:3032] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:16:09.777176 16996 raft_consensus.cc:729] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.777364 14886 leader_election.cc:288] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 2 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.777300 16847 consensus_queue.cc:237] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.777417 16996 consensus_queue.cc:260] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.777699 16996 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:16:09.778162 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.776621 14624 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.779750 14476 catalog_manager.cc:5484] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 1 to 2, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 2 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
W20240503 14:16:09.781535 16997 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20240503 14:16:09.782877 14625 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.784250 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.784564 14886 raft_consensus.cc:482] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.786381 14886 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.787662 15014 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 2.
I20240503 14:16:09.790249 14886 raft_consensus.cc:504] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.790885 14886 leader_election.cc:288] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.791410 14476 catalog_manager.cc:5484] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 1 to 2, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 2 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:09.791617 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.791812 14755 raft_consensus.cc:3032] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.791913 14755 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.791904 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.792085 14886 raft_consensus.cc:482] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.792178 14886 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.792171 14755 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.792512 14755 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.794726 14886 raft_consensus.cc:504] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.795339 14755 raft_consensus.cc:2396] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.796036 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.796264 14624 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.800650 14624 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.801072 14819 leader_election.cc:302] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.801409 16847 raft_consensus.cc:2781] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.801580 16847 raft_consensus.cc:686] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.801899 16847 consensus_queue.cc:237] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.802348 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.802572 14625 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.803232 14886 leader_election.cc:288] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.804220 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.804404 14755 raft_consensus.cc:3032] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.804494 14755 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.804716 14755 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.804981 14755 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.806977 14885 raft_consensus.cc:1261] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.807580 16997 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.810106 14476 catalog_manager.cc:5484] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.811704 14755 raft_consensus.cc:2396] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.811993 15014 raft_consensus.cc:1261] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.812762 14625 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.812822 16996 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.813088 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.813246 14886 raft_consensus.cc:482] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.813443 14886 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.813690 14819 leader_election.cc:302] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters: c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:09.816661 14886 raft_consensus.cc:504] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.817709 16310 raft_consensus.cc:2781] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.817924 16310 raft_consensus.cc:686] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.818486 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.818722 15016 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.823097 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.823354 14755 raft_consensus.cc:3032] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.823480 14755 raft_consensus.cc:729] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.823805 14755 consensus_queue.cc:260] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.824252 14755 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.818214 16310 consensus_queue.cc:237] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.818763 14886 leader_election.cc:288] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.829430 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.829720 14886 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.829854 14886 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.833602 14885 raft_consensus.cc:1261] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.833640 14886 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.834316 16996 consensus_queue.cc:1035] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.834857 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.835091 14756 raft_consensus.cc:3032] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.835177 14756 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.835450 14756 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.835731 14625 raft_consensus.cc:1261] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:09.835773 14756 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.835824 14625 pending_rounds.cc:76] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810: Aborting all ops after (but not including) 3
I20240503 14:16:09.835901 14625 pending_rounds.cc:97] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:09.836007 14625 raft_consensus.cc:2866] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:09.836710 17018 consensus_queue.cc:1035] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.838701 14886 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.839177 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.839607 14624 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.841127 14755 raft_consensus.cc:2396] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.842942 14756 raft_consensus.cc:2396] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.843355 14624 raft_consensus.cc:2396] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.843816 15016 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.844375 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.844663 14886 raft_consensus.cc:482] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.844847 14886 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.847119 14819 leader_election.cc:302] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:09.847543 16310 raft_consensus.cc:2781] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.848150 16310 raft_consensus.cc:686] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.848595 16310 consensus_queue.cc:237] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:09.847738 14819 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:09.850786 16310 raft_consensus.cc:2726] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:09.851254 14886 raft_consensus.cc:504] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.853404 14886 leader_election.cc:288] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.853899 15015 raft_consensus.cc:1261] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:09.854882 17014 consensus_queue.cc:1035] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.855595 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "843132c2c55645d598035edd0d2233c3"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:09.855931 14885 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.856034 14885 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.857739 14885 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.859015 14885 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:09.859630 14476 catalog_manager.cc:5484] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:09.860886 15014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.861114 15016 raft_consensus.cc:1261] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.861799 15014 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.861799 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:09.862032 15015 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.862804 16985 consensus_queue.cc:1035] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.863935 14885 raft_consensus.cc:1226] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Rejecting Update request from peer e9d4c39af431443189e8475561596810 for earlier term 2. Current term is 3. Ops: []
I20240503 14:16:09.864475 16997 consensus_queue.cc:1046] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.864698 16997 raft_consensus.cc:3032] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.864818 16997 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.864962 15014 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.865104 16997 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.865455 16997 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.865607 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.866207 14819 leader_election.cc:302] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:09.866174 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.866384 14624 raft_consensus.cc:3032] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:09.866529 16310 raft_consensus.cc:2781] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.866542 14624 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:09.866729 16310 raft_consensus.cc:686] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.866819 14624 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.866997 16310 consensus_queue.cc:237] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.867204 14624 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:09.868203 14484 catalog_manager.cc:5484] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:09.868868 15015 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.869278 14819 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:09.869546 17053 raft_consensus.cc:2781] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:09.869688 17053 raft_consensus.cc:686] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.869920 17053 consensus_queue.cc:237] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.870405 14624 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.871814 15016 raft_consensus.cc:1226] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Rejecting Update request from peer e9d4c39af431443189e8475561596810 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20240503 14:16:09.872370 14625 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 3.
I20240503 14:16:09.874877 14476 catalog_manager.cc:5484] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.875443 14476 catalog_manager.cc:5484] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 2 to 3, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 3 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:09.896745 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:09.897023 15013 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.897146 15013 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.898952 14625 raft_consensus.cc:1261] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.899533 17014 consensus_queue.cc:1035] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.899966 15013 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.901779 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.902066 14756 raft_consensus.cc:3032] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.902160 14756 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:09.902137 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.902379 14624 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.902534 14756 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.902992 14756 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.904980 14624 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:09.905336 14949 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:09.905529 15013 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:09.905611 17073 raft_consensus.cc:2781] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:09.906912 14756 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:09.907498 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:09.907757 15013 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.907896 15013 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.911667 15013 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.915257 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.915604 14885 raft_consensus.cc:3032] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.915712 14885 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.916105 14885 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:09.916507 14885 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.920596 15016 raft_consensus.cc:1261] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:09.921299 17014 consensus_queue.cc:1035] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.921584 15016 raft_consensus.cc:1261] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:09.923243 17014 consensus_queue.cc:1035] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.932075 14885 raft_consensus.cc:2396] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.936028 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:09.936343 14756 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.938549 15013 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:09.940923 14756 raft_consensus.cc:2396] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.941586 14950 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:09.942698 17099 raft_consensus.cc:2726] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:09.947597 14624 raft_consensus.cc:1261] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:09.947777 14624 pending_rounds.cc:76] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810: Aborting all ops after (but not including) 3
I20240503 14:16:09.947865 14624 pending_rounds.cc:97] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:09.947986 14624 raft_consensus.cc:2866] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:09.948809 17014 consensus_queue.cc:1035] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.957983 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "843132c2c55645d598035edd0d2233c3"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:09.958264 15013 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:09.958441 15013 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.960616 15014 raft_consensus.cc:1261] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.961395 17014 consensus_queue.cc:1035] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.971117 15012 raft_consensus.cc:1261] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.972101 17018 consensus_queue.cc:1035] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.981091 14625 raft_consensus.cc:1261] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:09.981226 14625 pending_rounds.cc:76] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810: Aborting all ops after (but not including) 3
I20240503 14:16:09.981287 14625 pending_rounds.cc:97] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:09.981362 14625 raft_consensus.cc:2866] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:09.982854 16585 consensus_queue.cc:1035] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.983191 17073 raft_consensus.cc:686] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:09.984429 15012 raft_consensus.cc:1261] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:09.984823 17073 consensus_queue.cc:237] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:09.985122 16585 consensus_queue.cc:1035] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.987064 14625 raft_consensus.cc:1261] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:09.987576 16585 consensus_queue.cc:1035] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.991528 15013 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.991921 14625 raft_consensus.cc:1261] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:09.992049 14625 pending_rounds.cc:76] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810: Aborting all ops after (but not including) 3
I20240503 14:16:09.992173 14625 pending_rounds.cc:97] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:09.992252 14625 raft_consensus.cc:2866] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:09.992761 16585 consensus_queue.cc:1035] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:09.995023 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:09.995239 14624 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:09.995606 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:09.995903 14885 raft_consensus.cc:3032] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:09.996031 14885 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:09.998283 14624 raft_consensus.cc:2396] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:09.998319 14885 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:09.999991 14885 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.003824 14885 raft_consensus.cc:2396] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.003981 15013 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.004567 14949 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810
I20240503 14:16:10.004844 17098 raft_consensus.cc:2726] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:10.006471 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.006819 15013 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.007033 15013 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.010151 15013 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.011276 14483 catalog_manager.cc:5484] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.011644 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.011852 14624 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.014761 14624 raft_consensus.cc:2396] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.016122 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.016393 14885 raft_consensus.cc:3032] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.016481 14885 raft_consensus.cc:729] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.016856 14885 consensus_queue.cc:260] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.017179 14885 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.020110 14885 raft_consensus.cc:2396] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.020475 14949 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810
I20240503 14:16:10.020794 17098 raft_consensus.cc:2726] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:10.020931 15013 leader_election.cc:288] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.022131 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.022370 15013 raft_consensus.cc:482] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.022521 15013 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.025223 15013 raft_consensus.cc:504] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.026139 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.026374 14624 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.026651 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.026845 14756 raft_consensus.cc:3032] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.026940 14756 raft_consensus.cc:729] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.027202 14756 consensus_queue.cc:260] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.027535 14756 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.029986 14624 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.030611 14756 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.030632 14949 leader_election.cc:302] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.031189 17098 raft_consensus.cc:2781] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:10.031329 17098 raft_consensus.cc:686] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.031646 17098 consensus_queue.cc:237] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.032768 15013 leader_election.cc:288] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.034917 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.035130 15013 raft_consensus.cc:482] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.035118 14624 raft_consensus.cc:1261] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.035216 14624 pending_rounds.cc:76] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810: Aborting all ops after (but not including) 3
I20240503 14:16:10.035236 15013 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.035317 14624 pending_rounds.cc:97] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.035398 14624 raft_consensus.cc:2866] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.035868 14479 catalog_manager.cc:5484] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 3 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.037259 16585 consensus_queue.cc:1035] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.037976 15013 raft_consensus.cc:504] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.038647 15013 leader_election.cc:288] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.039710 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.039906 14756 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.041249 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.041918 14885 raft_consensus.cc:3032] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.042610 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.042784 15013 raft_consensus.cc:482] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.042899 15013 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.042899 14885 raft_consensus.cc:729] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.043931 14885 consensus_queue.cc:260] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.045359 14885 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.045709 15013 raft_consensus.cc:504] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.046283 15013 leader_election.cc:288] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.046849 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.047077 14625 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.050637 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.050870 14886 raft_consensus.cc:3032] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.050971 14886 raft_consensus.cc:729] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.051292 14886 consensus_queue.cc:260] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.051785 14886 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.054584 14755 raft_consensus.cc:1261] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.054690 14755 pending_rounds.cc:76] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194: Aborting all ops after (but not including) 3
I20240503 14:16:10.054790 14755 pending_rounds.cc:97] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.054925 14755 raft_consensus.cc:2866] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.059172 15013 raft_consensus.cc:1261] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.059866 17018 consensus_queue.cc:1035] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.060248 16310 consensus_queue.cc:1035] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.061853 14625 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.061887 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.062093 15015 raft_consensus.cc:482] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.062283 15015 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.063192 14886 raft_consensus.cc:2396] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.065897 14756 raft_consensus.cc:2396] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.062278 14949 leader_election.cc:302] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.067095 17073 raft_consensus.cc:2781] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:10.067299 17073 raft_consensus.cc:686] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.067659 17073 consensus_queue.cc:237] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.069963 15015 raft_consensus.cc:504] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.070873 15016 raft_consensus.cc:1261] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.072549 15015 leader_election.cc:288] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.073113 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.073381 14756 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.073982 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.074262 15015 raft_consensus.cc:482] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.074426 15015 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.079720 15015 raft_consensus.cc:504] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.079789 16647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.071961 17053 consensus_queue.cc:1035] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20240503 14:16:10.072140 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.083745 14886 raft_consensus.cc:3032] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.084545 14886 raft_consensus.cc:729] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.084928 14886 consensus_queue.cc:260] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.085325 14886 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.089265 14624 raft_consensus.cc:1261] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.089376 14624 pending_rounds.cc:76] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810: Aborting all ops after (but not including) 3
I20240503 14:16:10.089452 14624 pending_rounds.cc:97] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.089545 14624 raft_consensus.cc:2866] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.090529 17018 consensus_queue.cc:1035] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.091547 14886 raft_consensus.cc:2396] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.070005 14885 raft_consensus.cc:2396] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.092368 14949 leader_election.cc:302] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:10.092715 17098 raft_consensus.cc:2726] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:10.094383 14479 catalog_manager.cc:5484] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 2 to 3, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 3 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.095757 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.095952 14625 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.098744 14625 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.099117 14949 leader_election.cc:302] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.099483 17098 raft_consensus.cc:2781] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:10.099658 17098 raft_consensus.cc:686] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.099789 15016 raft_consensus.cc:1261] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer c04ff0b9952046aeb4aec65ee5f6b194: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.099947 17098 consensus_queue.cc:237] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.100340 16585 consensus_queue.cc:1035] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.103948 15015 leader_election.cc:288] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.105165 14756 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.105415 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.105556 15015 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.105695 15015 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.106842 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.107134 14950 leader_election.cc:302] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters: b09f5cddfdf44e989f47a06c8da867f8
I20240503 14:16:10.107209 14886 raft_consensus.cc:3032] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.107388 14886 raft_consensus.cc:729] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.107439 17073 raft_consensus.cc:2781] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:10.107630 17073 raft_consensus.cc:686] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.107959 14886 consensus_queue.cc:260] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.108003 17073 consensus_queue.cc:237] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.108449 14886 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.112382 15015 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.113014 15015 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.113425 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.113672 14756 raft_consensus.cc:3032] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.113817 14756 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.114114 14756 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.114404 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.114598 14625 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.114619 14756 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.117588 14756 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.118022 14950 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.119393 17095 raft_consensus.cc:2781] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.119585 17095 raft_consensus.cc:686] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.119917 17095 consensus_queue.cc:237] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.120273 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.120478 15015 raft_consensus.cc:482] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.120602 15015 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.118103 15013 raft_consensus.cc:1261] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.122025 17053 consensus_queue.cc:1035] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.122784 16974 heartbeater.cc:503] Master 127.20.240.126:44017 was elected leader, sending a full tablet report...
I20240503 14:16:10.123297 15015 raft_consensus.cc:504] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.123912 15015 leader_election.cc:288] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.124850 14886 raft_consensus.cc:2396] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.119053 14625 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.126878 15016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.127084 15016 raft_consensus.cc:482] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.127027 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.127228 14756 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.127261 15016 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.128595 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.128719 14996 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:16:10.128863 14886 raft_consensus.cc:3032] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.130381 14756 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.130503 14886 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.130725 15016 raft_consensus.cc:504] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.130774 14886 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.131186 14886 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.133352 14886 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.131016 14950 leader_election.cc:302] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.133821 17073 raft_consensus.cc:2781] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.133999 17073 raft_consensus.cc:686] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.134256 17073 consensus_queue.cc:237] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.136162 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.136374 14625 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.142360 14625 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.142814 14949 leader_election.cc:302] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.144176 17073 raft_consensus.cc:2781] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.144433 17073 raft_consensus.cc:686] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.144713 17073 consensus_queue.cc:237] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.151484 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.151755 14755 raft_consensus.cc:3032] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.151862 14755 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.152132 14755 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.152570 14755 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.152812 15016 leader_election.cc:288] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.153913 14483 catalog_manager.cc:5484] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 2 to 3, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 3 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:10.163301 14755 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.166659 15016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.167526 15016 raft_consensus.cc:482] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.167732 15016 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.171989 15016 raft_consensus.cc:504] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.172610 14756 raft_consensus.cc:1261] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:10.175670 17053 consensus_queue.cc:1035] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.182165 16777 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.190596 14625 raft_consensus.cc:1261] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:10.191202 17053 consensus_queue.cc:1035] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.193173 15016 leader_election.cc:288] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.193553 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.193773 14886 raft_consensus.cc:3032] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.193842 14886 raft_consensus.cc:729] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.194088 14886 consensus_queue.cc:260] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.194667 14886 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.195531 14865 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:16:10.197247 15012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.197438 15012 raft_consensus.cc:482] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.197556 15012 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.200541 15012 raft_consensus.cc:504] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.201879 15012 leader_election.cc:288] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.203478 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.203694 14884 raft_consensus.cc:3032] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.203783 14884 raft_consensus.cc:729] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.204028 14884 consensus_queue.cc:260] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.204367 14884 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.204711 14735 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 14:16:10.207589 14884 raft_consensus.cc:2396] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.209118 15013 raft_consensus.cc:1261] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:16:10.210206 16310 consensus_queue.cc:1035] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.211640 14622 raft_consensus.cc:1261] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.211769 14622 pending_rounds.cc:76] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810: Aborting all ops after (but not including) 3
I20240503 14:16:10.211851 14622 pending_rounds.cc:97] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.211848 14886 raft_consensus.cc:2396] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.211920 14622 raft_consensus.cc:2866] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.213016 16310 consensus_queue.cc:1035] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.226444 14483 catalog_manager.cc:5484] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.226958 14483 catalog_manager.cc:5484] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.227396 14483 catalog_manager.cc:5484] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.231585 15012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.231904 15012 raft_consensus.cc:482] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.232087 15012 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.227799 14483 catalog_manager.cc:5484] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 2 to 3, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 3 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.235677 15012 raft_consensus.cc:504] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.239375 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.239373 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.239574 14755 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.239620 14885 raft_consensus.cc:3032] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.239693 14885 raft_consensus.cc:729] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.240113 14885 consensus_queue.cc:260] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.238879 15012 leader_election.cc:288] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.240391 14885 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.242399 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0184718c9fac4009b834726d661642f3"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.242921 14755 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.243422 14950 leader_election.cc:302] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.244566 14885 raft_consensus.cc:2396] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.245198 17073 raft_consensus.cc:2781] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:10.245476 17073 raft_consensus.cc:686] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.245738 17073 consensus_queue.cc:237] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.246379 16515 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.260505 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.260716 14624 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.260814 15012 raft_consensus.cc:1261] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 14:16:10.263720 14624 raft_consensus.cc:2396] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.264067 17053 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:10.266256 14949 leader_election.cc:302] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810
I20240503 14:16:10.266705 17098 raft_consensus.cc:2726] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:10.267050 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.267329 14624 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.273072 14625 raft_consensus.cc:1261] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 14:16:10.273701 14755 raft_consensus.cc:1261] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.273830 14755 pending_rounds.cc:76] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194: Aborting all ops after (but not including) 3
I20240503 14:16:10.273890 14755 pending_rounds.cc:97] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.273968 14755 raft_consensus.cc:2866] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.274158 17053 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:10.274772 17062 consensus_queue.cc:1035] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.278976 14622 raft_consensus.cc:1261] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.279462 17053 consensus_queue.cc:1035] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.270228 14624 raft_consensus.cc:2396] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.295487 14949 leader_election.cc:302] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810
I20240503 14:16:10.295938 17098 raft_consensus.cc:2726] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240503 14:16:10.296900 15013 raft_consensus.cc:482] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.297045 15013 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.299957 15013 raft_consensus.cc:504] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.300999 15013 leader_election.cc:288] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.301715 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.301913 14755 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.302951 14604 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.303930 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.304095 15015 raft_consensus.cc:482] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.304205 15015 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.304627 14476 catalog_manager.cc:5484] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 2 to 3, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 3 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.306649 15015 raft_consensus.cc:504] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.307551 15015 leader_election.cc:288] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.307871 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.308091 14886 raft_consensus.cc:3032] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.308192 14886 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.308444 14886 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.308759 14886 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.310797 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.310963 15015 raft_consensus.cc:482] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.311053 15015 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.312841 14755 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.313315 14950 leader_election.cc:302] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.313607 17073 raft_consensus.cc:2781] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:10.313777 17073 raft_consensus.cc:686] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.314052 17073 consensus_queue.cc:237] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.314314 14625 raft_consensus.cc:1261] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.314391 15015 raft_consensus.cc:504] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.314963 17053 consensus_queue.cc:1035] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.316109 15015 leader_election.cc:288] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.316541 14886 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.317032 14949 leader_election.cc:302] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:10.317051 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.317034 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.317226 14623 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.317245 14885 raft_consensus.cc:3032] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.317322 14885 raft_consensus.cc:729] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.317577 14885 consensus_queue.cc:260] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.317895 14885 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.318447 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.318640 14755 raft_consensus.cc:3032] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:16:10.318732 14755 raft_consensus.cc:729] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.319367 17073 raft_consensus.cc:2781] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.319550 14476 catalog_manager.cc:5484] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 2 to 3, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 3 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.320012 14623 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.320561 14885 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.320797 17073 raft_consensus.cc:686] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.321043 17073 consensus_queue.cc:237] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.321204 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.321386 14949 leader_election.cc:302] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.321620 17098 raft_consensus.cc:2781] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:16:10.321391 15015 raft_consensus.cc:482] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.321745 15015 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.321832 17098 raft_consensus.cc:686] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.322054 17098 consensus_queue.cc:237] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.318969 14755 consensus_queue.cc:260] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.323545 14755 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:16:10.325449 15015 raft_consensus.cc:504] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.326207 15015 leader_election.cc:288] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.326469 14623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.326473 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.326653 14623 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.326654 14756 raft_consensus.cc:3032] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.326752 14756 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.326984 14756 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.327332 14756 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.327961 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.328125 14624 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.329218 14623 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.329530 14949 leader_election.cc:302] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.330024 14756 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.330336 17098 raft_consensus.cc:2781] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.330361 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.330484 17098 raft_consensus.cc:686] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.330530 15015 raft_consensus.cc:482] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.330616 15015 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.330746 17098 consensus_queue.cc:237] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.333145 15015 raft_consensus.cc:504] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.334354 15015 leader_election.cc:288] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.334386 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.334578 14756 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.334853 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.335054 14885 raft_consensus.cc:3032] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.335127 14885 raft_consensus.cc:729] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.335397 14885 consensus_queue.cc:260] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.335676 14885 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.339031 14624 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.339141 14885 raft_consensus.cc:2396] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.339900 14756 raft_consensus.cc:2396] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.340714 14950 leader_election.cc:302] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:10.341181 17073 raft_consensus.cc:2726] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:10.341514 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.341606 14755 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 3.
I20240503 14:16:10.341692 15015 raft_consensus.cc:482] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.341799 15015 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.345296 16909 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:10.345124 14483 catalog_manager.cc:5484] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:10.345628 14483 catalog_manager.cc:5484] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.346117 14483 catalog_manager.cc:5484] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 2 to 3, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 3 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.348220 15015 raft_consensus.cc:504] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.349514 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.349786 14885 raft_consensus.cc:3032] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.349893 14885 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.353726 14885 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.354215 14885 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.355836 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.356110 14624 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.359174 14624 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.359776 14949 leader_election.cc:302] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.360106 17073 raft_consensus.cc:2781] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.360260 17073 raft_consensus.cc:686] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.360509 17073 consensus_queue.cc:237] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.361328 14755 raft_consensus.cc:1261] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Refusing update from remote peer b09f5cddfdf44e989f47a06c8da867f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.361513 14755 pending_rounds.cc:76] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194: Aborting all ops after (but not including) 3
I20240503 14:16:10.361654 14755 pending_rounds.cc:97] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.361812 14755 raft_consensus.cc:2866] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.361853 15015 leader_election.cc:288] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.362695 17053 consensus_queue.cc:1035] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.363750 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.363557 14476 catalog_manager.cc:5484] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.363951 15013 raft_consensus.cc:482] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.364061 15013 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.366948 15013 raft_consensus.cc:504] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.368047 15013 leader_election.cc:288] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.369036 15013 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:10.369210 15013 raft_consensus.cc:482] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.369338 15013 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.370342 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.370589 14755 raft_consensus.cc:3032] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.370715 14755 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.371001 14755 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.371402 14755 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.371933 15013 raft_consensus.cc:504] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.372776 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.373024 14625 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.374194 14755 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.374625 14950 leader_election.cc:302] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.374984 17073 raft_consensus.cc:2781] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.375219 17073 raft_consensus.cc:686] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.375569 17073 consensus_queue.cc:237] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.375783 14625 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.376456 15013 leader_election.cc:288] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.376804 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.376981 14624 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.377063 14754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.377249 14754 raft_consensus.cc:3032] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.377346 14754 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.377610 14754 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.377928 14754 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.380591 14885 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.380589 14754 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.381069 14950 leader_election.cc:302] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.381378 17097 raft_consensus.cc:2781] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.381528 17097 raft_consensus.cc:686] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.381832 17097 consensus_queue.cc:237] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.387630 14624 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 4.
I20240503 14:16:10.392513 14479 catalog_manager.cc:5484] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.393036 14479 catalog_manager.cc:5484] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 3 to 4, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 4 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.399235 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.399456 14624 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.399554 14624 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.402431 14624 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.403838 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.404063 14755 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.407047 14755 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.407474 14559 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.407794 16985 raft_consensus.cc:2781] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.407935 16985 raft_consensus.cc:686] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.408207 16985 consensus_queue.cc:237] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.410792 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.411041 15015 raft_consensus.cc:3032] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.411134 15015 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.411399 15015 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.411686 15015 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.414644 15015 raft_consensus.cc:2396] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.415679 14624 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.416882 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.417073 14624 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.417236 14624 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.416865 14476 catalog_manager.cc:5484] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.420526 14624 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.421573 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.421814 14885 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.424875 14885 raft_consensus.cc:2396] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.425969 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.426188 15015 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.426230 14624 leader_election.cc:288] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.427529 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.427805 14624 raft_consensus.cc:482] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.427971 14624 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.430002 15015 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.430599 14558 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters: b09f5cddfdf44e989f47a06c8da867f8
I20240503 14:16:10.431160 16997 raft_consensus.cc:2781] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.431099 14624 raft_consensus.cc:504] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.431378 16997 raft_consensus.cc:686] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.431732 16997 consensus_queue.cc:237] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.432229 14624 leader_election.cc:288] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.434224 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.434418 14624 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.434538 14624 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.434661 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.434911 15013 raft_consensus.cc:3032] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.435004 15013 raft_consensus.cc:729] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.435380 15013 consensus_queue.cc:260] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.435766 15013 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.437415 14624 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.438640 14624 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.438678 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.438923 14756 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.439338 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.439530 14755 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.439925 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.440117 14885 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.441699 14756 raft_consensus.cc:2396] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.441459 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46d6b19383c449198147cf7a999199d1"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.442160 14624 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.442361 14624 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.443159 15013 raft_consensus.cc:2396] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.445431 14624 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.446780 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.447116 14886 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.447392 14476 catalog_manager.cc:5484] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.451090 14886 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.451830 14624 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.451870 14558 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.451956 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.452134 14756 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.452255 16997 raft_consensus.cc:2781] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.452405 16997 raft_consensus.cc:686] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.452680 16997 consensus_queue.cc:237] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.454497 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "843132c2c55645d598035edd0d2233c3"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.454674 14624 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.454742 14756 raft_consensus.cc:2396] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.454793 14624 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.456660 14624 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.457146 14624 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.457916 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.458082 14624 raft_consensus.cc:482] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.458184 14624 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.458117 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.458420 14886 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.458858 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.459161 15013 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.459992 14624 raft_consensus.cc:504] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.460559 14624 leader_election.cc:288] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.461023 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.461023 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.461222 14756 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.461215 15012 raft_consensus.cc:3032] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.461295 15012 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.461571 15012 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.461892 15012 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.462558 14755 raft_consensus.cc:2454] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.464002 14756 raft_consensus.cc:2454] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.464661 15012 raft_consensus.cc:2396] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.465302 14885 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.465548 14559 leader_election.cc:302] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.465871 14558 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters: c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:10.466044 14559 leader_election.cc:302] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.466444 16985 raft_consensus.cc:2781] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.467913 14886 raft_consensus.cc:2396] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.468611 14476 catalog_manager.cc:5484] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.469523 16997 raft_consensus.cc:2781] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.470427 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.470592 14624 raft_consensus.cc:482] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.470705 14624 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.471689 15013 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.472222 14558 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters: b09f5cddfdf44e989f47a06c8da867f8
I20240503 14:16:10.474010 14624 raft_consensus.cc:504] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.474835 14624 leader_election.cc:288] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.475983 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.476213 14624 raft_consensus.cc:482] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.476336 14624 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.479976 14624 raft_consensus.cc:504] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.480808 14624 leader_election.cc:288] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.481957 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.482156 14624 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.482426 14624 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.485860 14624 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.486691 14624 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.486944 17144 raft_consensus.cc:2781] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.487614 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.487841 14624 raft_consensus.cc:482] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.487982 14624 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.488082 17146 raft_consensus.cc:2781] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.488211 17146 raft_consensus.cc:686] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.488528 17146 consensus_queue.cc:237] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.489179 16985 raft_consensus.cc:686] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.489466 16985 consensus_queue.cc:237] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.491571 14624 raft_consensus.cc:504] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.491678 17144 raft_consensus.cc:686] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.491986 17144 consensus_queue.cc:237] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.492173 14624 leader_election.cc:288] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.492540 14756 raft_consensus.cc:1261] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.492529 14476 catalog_manager.cc:5484] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 4, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 4 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.493259 17073 consensus_queue.cc:1035] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.493300 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.493840 14885 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.493911 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.494165 14886 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.497303 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.497454 14886 raft_consensus.cc:2396] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.497506 14624 raft_consensus.cc:482] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.497658 14624 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.498006 14885 raft_consensus.cc:2396] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.499344 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.500886 15012 raft_consensus.cc:3032] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.500829 14624 raft_consensus.cc:504] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.501014 15012 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.501336 15012 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.501547 14624 leader_election.cc:288] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.501619 15012 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.502565 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.502733 14625 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.502863 14625 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.503897 14885 raft_consensus.cc:1261] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.504104 14885 pending_rounds.cc:76] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8: Aborting all ops after (but not including) 3
I20240503 14:16:10.504257 14885 pending_rounds.cc:97] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.504468 14885 raft_consensus.cc:2866] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.504626 15012 raft_consensus.cc:2396] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.505062 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.505237 15015 raft_consensus.cc:3032] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.505342 15015 raft_consensus.cc:729] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.505458 17098 consensus_queue.cc:1035] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.505510 14625 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.505645 15015 consensus_queue.cc:260] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.505915 15015 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.506081 14625 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.507151 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.507342 14625 raft_consensus.cc:482] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.507452 14625 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.509047 15015 raft_consensus.cc:2396] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.509375 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.509549 15013 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.512041 15013 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.512830 14625 raft_consensus.cc:504] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.513442 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.513505 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.513646 14886 raft_consensus.cc:3032] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.513754 14886 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.514103 14884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.514060 14886 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.514283 14884 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.514521 14886 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.516192 14625 leader_election.cc:288] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.517094 14884 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.517786 14558 leader_election.cc:302] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.518113 14558 leader_election.cc:302] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8
I20240503 14:16:10.518752 14558 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.519527 14886 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.519546 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.519719 15015 raft_consensus.cc:3032] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.519827 15015 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.513648 14885 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.520190 14558 leader_election.cc:302] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.520553 17147 raft_consensus.cc:2781] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.520752 16985 raft_consensus.cc:2726] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:10.521556 16985 raft_consensus.cc:2781] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.521747 16985 raft_consensus.cc:686] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.520969 15015 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.522401 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.522578 14625 raft_consensus.cc:482] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.522706 14625 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.522742 15015 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.525475 15015 raft_consensus.cc:2396] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.522333 16985 consensus_queue.cc:237] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.526522 14885 raft_consensus.cc:2396] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.527132 16997 raft_consensus.cc:686] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.527284 17145 raft_consensus.cc:2781] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.527417 16997 consensus_queue.cc:237] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
W20240503 14:16:10.528579 14559 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.20.240.66:36573, user_credentials={real_user=slave}} blocked reactor thread for 62545.4us
I20240503 14:16:10.529510 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.529839 14756 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.530161 14625 raft_consensus.cc:504] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.530957 14625 leader_election.cc:288] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.531401 17145 raft_consensus.cc:686] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.531957 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.532212 14885 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.532123 17145 consensus_queue.cc:237] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.533298 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.533488 14625 raft_consensus.cc:482] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.533598 14625 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.534119 14756 raft_consensus.cc:2454] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.535189 14885 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.535622 14558 leader_election.cc:302] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.535650 14756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.529944 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.531674 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.535935 14756 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.535977 15012 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.538182 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.538403 14755 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.538408 16985 raft_consensus.cc:2781] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.539078 15012 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.539144 16985 raft_consensus.cc:686] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.539532 16985 consensus_queue.cc:237] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.539736 15013 raft_consensus.cc:3032] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.539844 15013 raft_consensus.cc:729] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.540306 15013 consensus_queue.cc:260] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.540585 15013 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.541492 17147 raft_consensus.cc:686] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.541822 17147 consensus_queue.cc:237] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.544006 14625 raft_consensus.cc:504] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.544657 14625 leader_election.cc:288] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.545248 15013 raft_consensus.cc:2396] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.546025 14558 leader_election.cc:302] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8
I20240503 14:16:10.546504 17151 raft_consensus.cc:2726] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240503 14:16:10.547751 14755 raft_consensus.cc:2396] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.547966 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.548255 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.548399 14885 raft_consensus.cc:3032] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.548527 14624 raft_consensus.cc:482] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.548679 14624 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.548830 14885 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.549311 14885 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.549983 14885 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.552191 14885 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.552666 14558 leader_election.cc:302] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.552744 14624 raft_consensus.cc:504] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.553105 17151 raft_consensus.cc:2781] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.553350 17151 raft_consensus.cc:686] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.553730 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.553925 15013 raft_consensus.cc:3032] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.553995 15013 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.553748 17151 consensus_queue.cc:237] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.554262 15013 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.554520 15013 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.557494 15013 raft_consensus.cc:2396] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.558085 14624 leader_election.cc:288] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.558751 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.559017 14885 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.559057 14625 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.559232 14625 raft_consensus.cc:482] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.559350 14625 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.562427 14625 raft_consensus.cc:504] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.563128 14625 leader_election.cc:288] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.564005 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.564203 14622 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.564314 14622 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.564610 14885 raft_consensus.cc:2454] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.565160 14558 leader_election.cc:302] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.565783 17145 raft_consensus.cc:2781] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.565987 17145 raft_consensus.cc:686] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.566356 17145 consensus_queue.cc:237] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.567528 14622 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.568143 14622 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.568377 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.568722 15013 raft_consensus.cc:3032] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.569351 14754 raft_consensus.cc:1261] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 14:16:10.569444 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.569628 14622 raft_consensus.cc:482] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.569747 14622 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.569993 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.570070 17098 consensus_queue.cc:1035] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:10.570325 14885 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.572410 14622 raft_consensus.cc:504] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.568874 15013 raft_consensus.cc:729] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.573433 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.573616 14886 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.573496 15013 consensus_queue.cc:260] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.573920 15013 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.576431 14886 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.577288 14558 leader_election.cc:302] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.577630 17151 raft_consensus.cc:2781] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.577766 17151 raft_consensus.cc:686] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.578032 17151 consensus_queue.cc:237] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.579187 14886 raft_consensus.cc:1261] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 14:16:10.579813 17073 consensus_queue.cc:1035] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:16:10.579919 14885 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.580426 14558 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.580924 14754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.581061 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.581187 14754 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.581250 14753 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.582473 14752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.582700 14752 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.585008 14754 raft_consensus.cc:2454] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.585863 14752 raft_consensus.cc:2396] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.587510 15013 raft_consensus.cc:2396] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.587966 14559 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.588356 17147 raft_consensus.cc:2781] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.588598 14559 leader_election.cc:302] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.588627 17147 raft_consensus.cc:686] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.588888 17144 raft_consensus.cc:2781] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.589040 17144 raft_consensus.cc:686] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.589092 17147 consensus_queue.cc:237] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.589244 14753 raft_consensus.cc:2454] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.589267 17144 consensus_queue.cc:237] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.589648 14756 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.589746 17145 raft_consensus.cc:2781] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.589900 17145 raft_consensus.cc:686] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.590215 17145 consensus_queue.cc:237] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.592913 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.593138 14755 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.593106 14483 catalog_manager.cc:5484] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.593690 14483 catalog_manager.cc:5484] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.594151 14483 catalog_manager.cc:5484] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.595873 14755 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 4.
I20240503 14:16:10.596796 14622 leader_election.cc:288] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.596873 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.597309 15013 raft_consensus.cc:3032] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.597458 15013 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.597895 14624 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.597862 15013 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.598100 14624 raft_consensus.cc:482] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.598369 14624 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.598505 15013 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.601806 14624 raft_consensus.cc:504] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.603057 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.603230 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.603313 15012 raft_consensus.cc:3032] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.603420 14755 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.603423 15012 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.604094 15012 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.604487 15012 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.606134 14559 leader_election.cc:302] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.606431 17151 raft_consensus.cc:2781] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.606654 17151 raft_consensus.cc:686] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.606918 17151 consensus_queue.cc:237] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.608093 15012 raft_consensus.cc:2396] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.608525 14624 leader_election.cc:288] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.609503 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.609704 14622 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.609799 14622 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.612151 15013 raft_consensus.cc:2396] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240503 14:16:10.612855 14483 catalog_manager.cc:5484] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.613360 14483 catalog_manager.cc:5484] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 4, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 4 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.613819 14483 catalog_manager.cc:5484] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.614316 14483 catalog_manager.cc:5484] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.615104 14483 catalog_manager.cc:5484] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:10.615727 14483 catalog_manager.cc:5484] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.616138 14483 catalog_manager.cc:5484] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 4, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 4 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.616526 14483 catalog_manager.cc:5484] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 2 to 4, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 4 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.616994 14483 catalog_manager.cc:5484] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.617435 14483 catalog_manager.cc:5484] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.621428 14755 raft_consensus.cc:2454] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.622524 14622 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.624059 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.624354 14885 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.628629 14885 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.629310 14558 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.629782 17147 raft_consensus.cc:2781] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.629962 17147 raft_consensus.cc:686] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.630409 17147 consensus_queue.cc:237] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.632143 14559 leader_election.cc:302] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.634406 17151 raft_consensus.cc:2781] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.634620 17151 raft_consensus.cc:686] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.634882 17151 consensus_queue.cc:237] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.636090 14622 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.636483 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.636711 14755 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.640475 14755 raft_consensus.cc:2396] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.641110 14622 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3"
dest_uuid: "e9d4c39af431443189e8475561596810"
from {username='slave'} at 127.0.0.1:50726
I20240503 14:16:10.641281 14622 raft_consensus.cc:482] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.641388 14622 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.644495 14622 raft_consensus.cc:504] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.646102 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.646584 14755 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.646975 14483 catalog_manager.cc:5484] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 4, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 4 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.649494 14483 catalog_manager.cc:5484] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 3 to 5, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.656014 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "e9d4c39af431443189e8475561596810" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.656232 15013 raft_consensus.cc:3032] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.656332 15013 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.656618 14622 leader_election.cc:288] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.656611 15013 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.656941 15013 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.656759 14483 catalog_manager.cc:5484] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.658743 14755 raft_consensus.cc:2454] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9d4c39af431443189e8475561596810 in term 5.
I20240503 14:16:10.659786 14559 leader_election.cc:302] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:10.660058 17147 raft_consensus.cc:2781] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.660336 17147 raft_consensus.cc:686] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Leader. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.660375 15013 raft_consensus.cc:2396] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e9d4c39af431443189e8475561596810 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.661326 17147 consensus_queue.cc:237] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.663887 14483 catalog_manager.cc:5484] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to e9d4c39af431443189e8475561596810 (127.20.240.65). New cstate: current_term: 5 leader_uuid: "e9d4c39af431443189e8475561596810" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.672995 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.673223 14755 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.673324 14755 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.675664 14755 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.677032 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.677172 14755 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.677246 15013 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.677639 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.677809 14622 raft_consensus.cc:3032] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.677867 14622 raft_consensus.cc:729] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.678117 14622 consensus_queue.cc:260] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.678392 14622 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.680051 15013 raft_consensus.cc:2396] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.680531 14622 raft_consensus.cc:2396] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.681064 14689 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.681392 16585 raft_consensus.cc:2726] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:10.681819 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.682089 14755 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.682296 14755 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.684094 14755 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.685053 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.685283 14885 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.685242 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.685432 15013 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.688266 14885 raft_consensus.cc:2396] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.688510 15013 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 5.
I20240503 14:16:10.688831 14755 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.688875 14689 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters: b09f5cddfdf44e989f47a06c8da867f8
I20240503 14:16:10.689219 16585 raft_consensus.cc:2781] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.689805 14756 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.690001 14756 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.690125 14756 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.692718 14756 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.693289 14756 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.695955 14756 raft_consensus.cc:1261] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240503 14:16:10.696452 17073 consensus_queue.cc:1035] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.695721 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.697367 14622 raft_consensus.cc:3032] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.697450 14622 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.697966 14622 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.698421 14622 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.698652 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.698858 14885 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.704182 14885 raft_consensus.cc:2454] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 6.
I20240503 14:16:10.704769 14689 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.710757 14622 raft_consensus.cc:2396] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.711970 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46d6b19383c449198147cf7a999199d1"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.712226 14755 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.712348 14755 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.720206 14755 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.722800 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.723055 14885 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.727022 14885 raft_consensus.cc:2454] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 6.
I20240503 14:16:10.728418 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.728713 14622 raft_consensus.cc:3032] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.728819 14622 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.729231 14622 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.729651 14622 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.733909 14622 raft_consensus.cc:2396] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.721042 14755 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.721258 17165 raft_consensus.cc:2781] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:10.734695 17165 raft_consensus.cc:686] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.735008 17165 consensus_queue.cc:237] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.735721 14755 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.735900 14755 raft_consensus.cc:482] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.721649 16585 raft_consensus.cc:686] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.736073 14755 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.736323 16585 consensus_queue.cc:237] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.732826 14689 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.737556 17168 raft_consensus.cc:2781] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:10.737697 17168 raft_consensus.cc:686] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.738014 17168 consensus_queue.cc:237] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.740236 14755 raft_consensus.cc:504] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.741509 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.741825 14622 raft_consensus.cc:3032] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.741933 14622 raft_consensus.cc:729] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.742509 14755 leader_election.cc:288] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.742290 14622 consensus_queue.cc:260] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.743039 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e84474717eb440f3ad48c19a408a467b" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.743175 14622 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.743255 15013 raft_consensus.cc:3037] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.746086 15013 raft_consensus.cc:2396] T e84474717eb440f3ad48c19a408a467b P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.746126 14622 raft_consensus.cc:2396] T e84474717eb440f3ad48c19a408a467b P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.747552 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.747771 14753 raft_consensus.cc:482] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.747885 14753 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.747695 14483 catalog_manager.cc:5484] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 3 to 5, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 5 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.748085 14689 leader_election.cc:302] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.749042 14885 raft_consensus.cc:1261] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Refusing update from remote peer 49e64758760942e1abdcdc0d2e1f07dd: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240503 14:16:10.749161 14885 pending_rounds.cc:76] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8: Aborting all ops after (but not including) 3
I20240503 14:16:10.749236 14885 pending_rounds.cc:97] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240503 14:16:10.749302 14885 raft_consensus.cc:2866] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.749920 17097 consensus_queue.cc:1035] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.750473 14753 raft_consensus.cc:504] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.751102 14753 leader_election.cc:288] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.752213 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.752209 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.752386 15013 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.752395 14886 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.755082 15013 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 5.
I20240503 14:16:10.755245 14886 raft_consensus.cc:2396] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.755892 14689 leader_election.cc:302] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.756054 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.756211 14753 raft_consensus.cc:482] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.756327 14753 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.756407 17166 raft_consensus.cc:2726] T e84474717eb440f3ad48c19a408a467b P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 14:16:10.756531 17166 raft_consensus.cc:2781] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.756682 17166 raft_consensus.cc:686] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.756984 17166 consensus_queue.cc:237] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.759517 14753 raft_consensus.cc:504] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.760649 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.760895 14886 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.764180 14886 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 5.
I20240503 14:16:10.764550 14689 leader_election.cc:302] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.764925 17168 raft_consensus.cc:2781] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.765092 17168 raft_consensus.cc:686] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.765316 17168 consensus_queue.cc:237] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.765455 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.765661 15013 raft_consensus.cc:3032] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.765735 15013 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.765990 15013 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.766347 15013 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.770097 15013 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 5.
I20240503 14:16:10.771579 14753 leader_election.cc:288] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.774106 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.774317 14753 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.774422 14753 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.775092 14483 catalog_manager.cc:5484] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 6 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.775516 14483 catalog_manager.cc:5484] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 3 to 5, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 5 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.775882 14483 catalog_manager.cc:5484] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 6 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.777079 14753 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.778100 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.778323 14622 raft_consensus.cc:3032] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.778398 14622 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.778681 14622 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.779424 14622 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.780973 14483 catalog_manager.cc:5484] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 4 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 5 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.782225 14622 raft_consensus.cc:2396] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.783208 14753 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.783908 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.784108 14885 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.784206 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.784380 14753 raft_consensus.cc:482] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.784487 14753 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.785948 14885 raft_consensus.cc:2454] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 6.
I20240503 14:16:10.786330 14689 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:10.786619 17166 raft_consensus.cc:2781] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:10.786772 17166 raft_consensus.cc:686] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.787007 17166 consensus_queue.cc:237] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.787187 14753 raft_consensus.cc:504] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.787847 14753 leader_election.cc:288] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.788667 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.788874 15013 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.789149 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceccdb68dda4765ad1c8e4051d2e4c1" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.789400 14622 raft_consensus.cc:3032] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.789510 14622 raft_consensus.cc:729] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.789973 14622 consensus_queue.cc:260] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.790391 14622 raft_consensus.cc:3037] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.791587 15013 raft_consensus.cc:2396] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.792176 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.792357 14753 raft_consensus.cc:482] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.792467 14753 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.794292 14622 raft_consensus.cc:2396] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.794759 14689 leader_election.cc:302] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.795178 16585 raft_consensus.cc:2726] T 4ceccdb68dda4765ad1c8e4051d2e4c1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:10.796815 14753 raft_consensus.cc:504] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.797951 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.798156 14885 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.798451 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.798697 15013 raft_consensus.cc:3032] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.798833 15013 raft_consensus.cc:729] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.799160 15013 consensus_queue.cc:260] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.799659 15013 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.800712 14885 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 4.
I20240503 14:16:10.802692 14689 leader_election.cc:302] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.803202 16585 raft_consensus.cc:2781] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.803346 16585 raft_consensus.cc:686] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.803570 16585 consensus_queue.cc:237] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.805295 15013 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 4.
I20240503 14:16:10.805886 14753 leader_election.cc:288] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.805903 14483 catalog_manager.cc:5484] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 6 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.807139 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0184718c9fac4009b834726d661642f3"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.807298 14753 raft_consensus.cc:482] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.807405 14753 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.810035 14753 raft_consensus.cc:504] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.810673 14753 leader_election.cc:288] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 4 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.811434 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.811622 14753 raft_consensus.cc:482] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.811723 14753 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.812723 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.812867 15013 raft_consensus.cc:3032] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.812958 15013 raft_consensus.cc:729] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.813215 15013 consensus_queue.cc:260] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.813452 15013 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.813483 14483 catalog_manager.cc:5484] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 3 to 4, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 4 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.814621 14753 raft_consensus.cc:504] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.815248 14753 leader_election.cc:288] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.815354 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.815567 14885 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.815722 15013 raft_consensus.cc:2396] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.816720 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.816923 15013 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.817301 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aab5902c7cd4daaada285f05615dfad" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.817509 14622 raft_consensus.cc:3032] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.817593 14622 raft_consensus.cc:729] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.817852 14622 consensus_queue.cc:260] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.818079 14885 raft_consensus.cc:2454] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 4.
I20240503 14:16:10.818168 14622 raft_consensus.cc:3037] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.818950 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.819131 14753 raft_consensus.cc:482] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.819161 14689 leader_election.cc:302] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:10.819259 14753 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.819455 17166 raft_consensus.cc:2781] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.819617 17166 raft_consensus.cc:686] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.819854 17166 consensus_queue.cc:237] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.821955 15013 raft_consensus.cc:2396] T 3aab5902c7cd4daaada285f05615dfad P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.822569 14476 catalog_manager.cc:5484] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 3 to 4, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 4 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.823230 14753 raft_consensus.cc:504] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.823781 14753 leader_election.cc:288] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.824409 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.824601 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46120f9b009a4e3d96143bd7cdca9f92" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.824785 15013 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.825115 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.825354 14753 raft_consensus.cc:482] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.825505 14753 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.825592 14624 raft_consensus.cc:3032] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.825695 14624 raft_consensus.cc:729] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.825954 14624 consensus_queue.cc:260] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.826179 14624 raft_consensus.cc:3037] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.827736 15013 raft_consensus.cc:2396] T 46120f9b009a4e3d96143bd7cdca9f92 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.828097 14624 raft_consensus.cc:2396] T 46120f9b009a4e3d96143bd7cdca9f92 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.828285 14753 raft_consensus.cc:504] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.828478 14689 leader_election.cc:302] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.828744 17166 raft_consensus.cc:2726] T 46120f9b009a4e3d96143bd7cdca9f92 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:10.829006 14753 leader_election.cc:288] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 4 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.829237 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.829411 15013 raft_consensus.cc:3032] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:16:10.829509 15013 raft_consensus.cc:729] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:10.829811 15013 consensus_queue.cc:260] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.830058 14622 raft_consensus.cc:2396] T 3aab5902c7cd4daaada285f05615dfad P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.830206 15013 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.830124 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.830855 14885 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:16:10.833428 15013 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 4.
I20240503 14:16:10.833606 14885 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 4.
I20240503 14:16:10.830695 14689 leader_election.cc:302] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.830360 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.834600 14753 raft_consensus.cc:482] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.834762 14753 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.834811 14689 leader_election.cc:302] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.835264 17166 raft_consensus.cc:2726] T 3aab5902c7cd4daaada285f05615dfad P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 14:16:10.835448 17166 raft_consensus.cc:2781] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:16:10.835585 17166 raft_consensus.cc:686] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.835846 17166 consensus_queue.cc:237] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.838723 14476 catalog_manager.cc:5484] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 3 to 4, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 4 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.839151 14753 raft_consensus.cc:504] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.839938 14753 leader_election.cc:288] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.841082 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.841114 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26c1a7f75f214911979a2fb26245c814" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.841272 15013 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.841291 14622 raft_consensus.cc:3032] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.841363 14622 raft_consensus.cc:729] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.841635 14622 consensus_queue.cc:260] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.841923 14622 raft_consensus.cc:3037] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.843164 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.843389 14753 raft_consensus.cc:482] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.843508 14753 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.844166 15013 raft_consensus.cc:2396] T 26c1a7f75f214911979a2fb26245c814 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.844777 14622 raft_consensus.cc:2396] T 26c1a7f75f214911979a2fb26245c814 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.845197 14689 leader_election.cc:302] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.845579 17168 raft_consensus.cc:2726] T 26c1a7f75f214911979a2fb26245c814 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:10.846524 14753 raft_consensus.cc:504] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.847530 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.847733 14885 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.847710 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.847898 14622 raft_consensus.cc:3032] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.847967 14622 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.848245 14622 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.848510 14622 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.850328 14753 leader_election.cc:288] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.850445 14885 raft_consensus.cc:2454] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 5.
I20240503 14:16:10.851193 14689 leader_election.cc:302] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.851274 14622 raft_consensus.cc:2396] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.851282 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.851449 14753 raft_consensus.cc:482] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.851481 17168 raft_consensus.cc:2781] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.851665 14753 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.851783 17168 raft_consensus.cc:686] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.852097 17168 consensus_queue.cc:237] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.854511 14753 raft_consensus.cc:504] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.855515 14476 catalog_manager.cc:5484] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 4 to 5, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 5 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.856863 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.857088 15013 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.857245 14753 leader_election.cc:288] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.858726 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.858906 14753 raft_consensus.cc:482] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.859000 14753 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.859680 15013 raft_consensus.cc:2396] T 340fdb957a4349c29a45e66ddb303104 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.861807 14753 raft_consensus.cc:504] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.862064 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "340fdb957a4349c29a45e66ddb303104" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.862284 14622 raft_consensus.cc:3032] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.862390 14622 raft_consensus.cc:729] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.862392 14753 leader_election.cc:288] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.862648 14622 consensus_queue.cc:260] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.862929 14622 raft_consensus.cc:3037] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.863293 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.863281 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.863368 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.863485 14624 raft_consensus.cc:3032] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:10.863442 14753 raft_consensus.cc:482] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.863586 14885 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.863581 14624 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.863617 14753 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.863862 14624 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.864154 14624 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.866070 14622 raft_consensus.cc:2396] T 340fdb957a4349c29a45e66ddb303104 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.866217 14885 raft_consensus.cc:2454] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 5.
I20240503 14:16:10.866652 14689 leader_election.cc:302] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.866948 14624 raft_consensus.cc:2396] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.866968 16585 raft_consensus.cc:2781] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.867110 16585 raft_consensus.cc:686] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.867362 16585 consensus_queue.cc:237] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.867525 14689 leader_election.cc:302] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.868079 16585 raft_consensus.cc:2726] T 340fdb957a4349c29a45e66ddb303104 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:10.869154 14753 raft_consensus.cc:504] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.869725 14753 leader_election.cc:288] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:10.870002 14476 catalog_manager.cc:5484] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 4 to 5, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 5 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.870811 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.870993 14753 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.871104 14753 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.871315 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.871520 15013 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.873690 14753 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.874300 15013 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 5.
I20240503 14:16:10.874706 14886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.874902 14886 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.874943 14689 leader_election.cc:302] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.875245 16585 raft_consensus.cc:2781] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:10.875404 16585 raft_consensus.cc:686] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.875650 16585 consensus_queue.cc:237] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.876471 14753 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.876720 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.876890 14624 raft_consensus.cc:3032] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.876942 14624 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.877226 14624 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.877427 14624 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.877562 14886 raft_consensus.cc:2454] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 6.
I20240503 14:16:10.878336 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.878508 14885 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:10.878724 14689 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:10.879001 17168 raft_consensus.cc:2781] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:10.879134 17168 raft_consensus.cc:686] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.879395 17168 consensus_queue.cc:237] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.879783 14624 raft_consensus.cc:2396] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.882853 14885 raft_consensus.cc:2396] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:10.885933 14483 catalog_manager.cc:5484] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 3 to 5, leader changed from b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 5 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:10.886196 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.886377 14753 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.886502 14753 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.887861 15013 raft_consensus.cc:1261] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:16:10.888442 17147 consensus_queue.cc:1035] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.888973 14753 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.889564 14753 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.890671 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.890913 14624 raft_consensus.cc:3032] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.891012 14624 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.891368 14624 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.891700 14624 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.895025 14753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3"
dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
from {username='slave'} at 127.0.0.1:54930
I20240503 14:16:10.895203 14624 raft_consensus.cc:2396] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.895242 14753 raft_consensus.cc:482] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.895363 14753 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.895852 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:10.896112 14885 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.899379 14483 catalog_manager.cc:5484] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 6 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:10.900950 14753 raft_consensus.cc:504] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.901897 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.902004 14885 raft_consensus.cc:2454] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c04ff0b9952046aeb4aec65ee5f6b194 in term 6.
I20240503 14:16:10.902118 14622 raft_consensus.cc:3032] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.902202 14622 raft_consensus.cc:729] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.902460 14622 consensus_queue.cc:260] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.902716 14622 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.904538 14689 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194; no voters: e9d4c39af431443189e8475561596810
I20240503 14:16:10.904996 17168 raft_consensus.cc:2781] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:10.905158 17168 raft_consensus.cc:686] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Leader. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.905376 17168 consensus_queue.cc:237] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.909229 14622 raft_consensus.cc:2396] T aab2e8db77094d73948bed276f95f8d3 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.910691 14753 leader_election.cc:288] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.910980 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aab2e8db77094d73948bed276f95f8d3" candidate_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.911180 15012 raft_consensus.cc:3037] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.913563 15012 raft_consensus.cc:2396] T aab2e8db77094d73948bed276f95f8d3 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c04ff0b9952046aeb4aec65ee5f6b194 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.914301 14885 raft_consensus.cc:1261] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:16:10.914660 14689 leader_election.cc:302] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c04ff0b9952046aeb4aec65ee5f6b194; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:10.914916 16585 raft_consensus.cc:2726] T aab2e8db77094d73948bed276f95f8d3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:10.915026 17151 consensus_queue.cc:1035] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.915331 17151 consensus_queue.cc:1225] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [LEADER]: Peer b09f5cddfdf44e989f47a06c8da867f8 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240503 14:16:10.916605 14885 pending_rounds.cc:76] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8: Aborting all ops after (but not including) 3
I20240503 14:16:10.916730 14885 pending_rounds.cc:97] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240503 14:16:10.916822 14885 raft_consensus.cc:2866] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.921723 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:10.921900 14886 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.921989 14886 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.924721 14886 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.926014 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.926227 15012 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.929024 15012 raft_consensus.cc:2454] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:10.929571 14819 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:10.929829 17053 raft_consensus.cc:2781] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:10.931149 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.931339 14753 raft_consensus.cc:3032] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.931437 14753 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.931677 14753 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.931977 14753 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.934957 14753 raft_consensus.cc:2396] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.935654 14886 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:10.938287 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:10.938450 14886 raft_consensus.cc:482] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.938552 14886 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:10.940049 17053 raft_consensus.cc:686] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.940320 17053 consensus_queue.cc:237] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:10.943454 14886 raft_consensus.cc:504] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.945070 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.945271 14753 raft_consensus.cc:3032] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:16:10.945364 14753 raft_consensus.cc:729] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.945616 14753 consensus_queue.cc:260] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.945895 14753 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:10.948760 14753 raft_consensus.cc:2396] T 1e04912cfaf1494295de9f9969fc3c58 P c04ff0b9952046aeb4aec65ee5f6b194 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.949923 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e04912cfaf1494295de9f9969fc3c58" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.950165 14622 raft_consensus.cc:3037] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:10.951306 14483 catalog_manager.cc:5484] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.952925 14622 raft_consensus.cc:2396] T 1e04912cfaf1494295de9f9969fc3c58 P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.953374 14819 leader_election.cc:302] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:10.953742 17053 raft_consensus.cc:2726] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 14:16:10.954015 14886 leader_election.cc:288] T 1e04912cfaf1494295de9f9969fc3c58 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.956512 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46d6b19383c449198147cf7a999199d1"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:10.957088 14886 raft_consensus.cc:482] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.957245 14886 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:10.957068 14479 catalog_manager.cc:5484] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66). New cstate: current_term: 6 leader_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:10.959954 14886 raft_consensus.cc:504] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.960945 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.961182 14753 raft_consensus.cc:3032] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:16:10.961315 14753 raft_consensus.cc:729] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.961618 14753 consensus_queue.cc:260] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:10.961966 14753 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:10.964855 14753 raft_consensus.cc:2396] T 46d6b19383c449198147cf7a999199d1 P c04ff0b9952046aeb4aec65ee5f6b194 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.965975 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d6b19383c449198147cf7a999199d1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.966168 14622 raft_consensus.cc:3037] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:10.968922 14622 raft_consensus.cc:2396] T 46d6b19383c449198147cf7a999199d1 P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:10.969305 14885 raft_consensus.cc:1261] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240503 14:16:10.969409 14885 pending_rounds.cc:76] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8: Aborting all ops after (but not including) 4
I20240503 14:16:10.969487 14819 leader_election.cc:302] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:10.969808 14885 pending_rounds.cc:97] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240503 14:16:10.969888 14885 raft_consensus.cc:2866] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:10.970369 17053 raft_consensus.cc:2726] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 14:16:10.970433 17151 consensus_queue.cc:1035] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.970649 14886 leader_election.cc:288] T 46d6b19383c449198147cf7a999199d1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:10.971658 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:10.971829 14885 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.971959 14885 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.974653 14885 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.975320 14885 leader_election.cc:288] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.976037 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.976123 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.976305 14622 raft_consensus.cc:3032] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.976320 15012 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.976385 14622 raft_consensus.cc:729] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.976656 14622 consensus_queue.cc:260] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.976995 14622 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.979130 15012 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:10.979617 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:10.979619 14819 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:10.979902 14885 raft_consensus.cc:482] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.980070 17053 raft_consensus.cc:2781] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:10.980193 17053 raft_consensus.cc:686] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:10.980293 14885 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.980475 17053 consensus_queue.cc:237] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.982023 15012 raft_consensus.cc:1261] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:16:10.982568 17151 consensus_queue.cc:1035] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:10.987617 14885 raft_consensus.cc:504] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.988626 14885 leader_election.cc:288] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.988473 14483 catalog_manager.cc:5484] T 435b95c19dfe45ceb5b15470a2ebd35e P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:10.989506 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:10.989676 14885 raft_consensus.cc:482] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.989799 14885 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.991000 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:10.991240 15012 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.992368 14885 raft_consensus.cc:504] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.993014 14885 leader_election.cc:288] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.993562 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:10.993774 14624 raft_consensus.cc:3032] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.993857 14624 raft_consensus.cc:729] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:10.993841 14884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:10.994006 14884 raft_consensus.cc:482] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:10.994112 14884 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.994122 14624 consensus_queue.cc:260] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:10.993433 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd6bbdd8c1d84a6d88e4810b0f125cf5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.994411 14624 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.994493 14753 raft_consensus.cc:3032] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.994599 14753 raft_consensus.cc:729] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.994858 14753 consensus_queue.cc:260] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.995184 14753 raft_consensus.cc:3037] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.996650 14884 raft_consensus.cc:504] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.997159 14624 raft_consensus.cc:2396] T c32b296d9dda42bd81f5e7aa1edc5774 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.997299 14884 leader_election.cc:288] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:10.998011 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:10.998219 14755 raft_consensus.cc:3032] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:10.998332 14755 raft_consensus.cc:729] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:10.998576 14755 consensus_queue.cc:260] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:10.998821 14755 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:10.999006 14753 raft_consensus.cc:2396] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:10.999534 14885 raft_consensus.cc:1261] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 14:16:10.999965 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0cc940dc9e4511856fecf6e43028e3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.000159 15016 raft_consensus.cc:3037] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.001396 14755 raft_consensus.cc:2396] T 4d0cc940dc9e4511856fecf6e43028e3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 14:16:11.001716 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32b296d9dda42bd81f5e7aa1edc5774" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.001917 15015 raft_consensus.cc:3037] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.002923 15016 raft_consensus.cc:2454] T 4d0cc940dc9e4511856fecf6e43028e3 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.003815 17146 consensus_queue.cc:1035] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:11.003867 14819 leader_election.cc:302] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:11.004209 17053 raft_consensus.cc:2781] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:11.004348 17053 raft_consensus.cc:686] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.004566 17053 consensus_queue.cc:237] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.004679 15015 raft_consensus.cc:2396] T c32b296d9dda42bd81f5e7aa1edc5774 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.006758 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.006953 14885 raft_consensus.cc:482] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.007094 14885 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.008216 14819 leader_election.cc:302] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810
I20240503 14:16:11.008530 17053 raft_consensus.cc:2726] T c32b296d9dda42bd81f5e7aa1edc5774 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:11.016957 14885 raft_consensus.cc:504] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.017899 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.018136 14624 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.018435 15012 raft_consensus.cc:2454] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.018862 14819 leader_election.cc:302] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:11.019196 14622 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.019292 17053 raft_consensus.cc:2781] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:11.019440 17053 raft_consensus.cc:686] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.019707 17053 consensus_queue.cc:237] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.020992 14624 raft_consensus.cc:2396] T c8c2fc935476474182cf1bf00c15b547 P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.022611 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8c2fc935476474182cf1bf00c15b547" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.022804 14755 raft_consensus.cc:3032] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:16:11.022905 14755 raft_consensus.cc:729] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.023196 14755 consensus_queue.cc:260] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.023487 14755 raft_consensus.cc:3037] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.025879 14755 raft_consensus.cc:2396] T c8c2fc935476474182cf1bf00c15b547 P c04ff0b9952046aeb4aec65ee5f6b194 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.026350 14820 leader_election.cc:302] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:11.026639 17201 raft_consensus.cc:2726] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 14:16:11.029883 14885 leader_election.cc:288] T c8c2fc935476474182cf1bf00c15b547 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.032788 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.034217 14885 raft_consensus.cc:482] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.034442 14885 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.036243 14886 raft_consensus.cc:1261] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240503 14:16:11.036819 17151 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.037550 14885 raft_consensus.cc:504] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.037897 14483 catalog_manager.cc:5484] T 4d0cc940dc9e4511856fecf6e43028e3 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:11.038620 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.038574 14483 catalog_manager.cc:5484] T dd6bbdd8c1d84a6d88e4810b0f125cf5 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:11.039044 14885 leader_election.cc:288] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:11.039810 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72b8519d872146b98f942fe73c5e5e84" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.040047 15012 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.040897 15015 raft_consensus.cc:1261] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240503 14:16:11.041026 15015 pending_rounds.cc:76] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd: Aborting all ops after (but not including) 5
I20240503 14:16:11.041110 15015 pending_rounds.cc:97] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240503 14:16:11.041280 15015 raft_consensus.cc:2866] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:11.042624 17146 consensus_queue.cc:1035] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:16:11.042842 15012 raft_consensus.cc:2454] T 72b8519d872146b98f942fe73c5e5e84 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 5.
I20240503 14:16:11.043814 14819 leader_election.cc:302] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:11.044382 17201 raft_consensus.cc:2781] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:11.044569 17201 raft_consensus.cc:686] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.044834 17201 consensus_queue.cc:237] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.048410 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0184718c9fac4009b834726d661642f3"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.048666 14755 raft_consensus.cc:3032] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:11.048806 14755 raft_consensus.cc:729] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.049130 14755 consensus_queue.cc:260] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.049489 14755 raft_consensus.cc:3037] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.049345 14483 catalog_manager.cc:5484] T 72b8519d872146b98f942fe73c5e5e84 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 4 to 5, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 5 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:11.052568 14755 raft_consensus.cc:2396] T 72b8519d872146b98f942fe73c5e5e84 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.053372 14885 raft_consensus.cc:482] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.053552 14885 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.054402 14883 raft_consensus.cc:1261] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240503 14:16:11.054514 14883 pending_rounds.cc:76] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8: Aborting all ops after (but not including) 4
I20240503 14:16:11.054600 14883 pending_rounds.cc:97] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240503 14:16:11.054670 14883 raft_consensus.cc:2866] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240503 14:16:11.056586 14885 raft_consensus.cc:504] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.057796 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.058024 14755 raft_consensus.cc:3032] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:11.058120 14755 raft_consensus.cc:729] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.058432 14755 consensus_queue.cc:260] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.058763 14755 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.059752 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0184718c9fac4009b834726d661642f3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.059752 17146 consensus_queue.cc:1035] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:11.059964 15015 raft_consensus.cc:3037] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.061640 14755 raft_consensus.cc:2396] T 0184718c9fac4009b834726d661642f3 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:11.062144 14885 leader_election.cc:288] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.063517 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.063736 14885 raft_consensus.cc:482] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.063865 14885 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.063949 15012 raft_consensus.cc:1261] T 67cbe83d27cc4107b88da1f87c5468e2 P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240503 14:16:11.064538 17146 consensus_queue.cc:1035] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:11.064836 15015 raft_consensus.cc:2396] T 0184718c9fac4009b834726d661642f3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:11.065459 14819 leader_election.cc:302] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:11.066212 17053 raft_consensus.cc:2726] T 0184718c9fac4009b834726d661642f3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240503 14:16:11.066977 14885 raft_consensus.cc:504] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.068039 14883 raft_consensus.cc:1226] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Rejecting Update request from peer e9d4c39af431443189e8475561596810 for earlier term 4. Current term is 5. Ops: []
I20240503 14:16:11.068405 14885 leader_election.cc:288] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:11.068856 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.069039 17147 consensus_queue.cc:1046] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:16:11.069404 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.069562 14885 raft_consensus.cc:482] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.069584 14624 raft_consensus.cc:3032] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:11.069684 14624 raft_consensus.cc:729] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:11.069697 14885 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.069945 14624 consensus_queue.cc:260] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.070284 14624 raft_consensus.cc:3037] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.072448 14885 raft_consensus.cc:504] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.072952 14624 raft_consensus.cc:2454] T 67cbe83d27cc4107b88da1f87c5468e2 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 5.
W20240503 14:16:11.073082 17145 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20240503 14:16:11.073519 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.073720 14755 raft_consensus.cc:3032] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:16:11.073804 14755 raft_consensus.cc:729] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.073829 14819 leader_election.cc:302] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:11.073915 15016 raft_consensus.cc:1261] T d24bc6dd77904c7e9580939b27641500 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Refusing update from remote peer e9d4c39af431443189e8475561596810: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240503 14:16:11.074062 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34c5b61b82b34beb8d10677e24e6f418" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.074079 14755 consensus_queue.cc:260] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.074214 14624 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.074342 17053 raft_consensus.cc:2781] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:11.074383 14755 raft_consensus.cc:3037] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.074508 17053 raft_consensus.cc:686] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.074738 17053 consensus_queue.cc:237] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.075659 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67cbe83d27cc4107b88da1f87c5468e2" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.076452 17145 consensus_queue.cc:1035] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 14:16:11.077250 14755 raft_consensus.cc:2396] T 34c5b61b82b34beb8d10677e24e6f418 P c04ff0b9952046aeb4aec65ee5f6b194 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.078486 14479 catalog_manager.cc:5484] T 67cbe83d27cc4107b88da1f87c5468e2 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 4 to 5, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 5 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:11.078950 14885 leader_election.cc:288] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.080011 14883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.080278 14883 raft_consensus.cc:482] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.080391 14883 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.082454 14624 raft_consensus.cc:2396] T 34c5b61b82b34beb8d10677e24e6f418 P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.082944 14883 raft_consensus.cc:504] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.083547 14883 leader_election.cc:288] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:11.084558 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.084700 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.084777 14622 raft_consensus.cc:3032] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:11.084846 14622 raft_consensus.cc:729] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:11.084890 14886 raft_consensus.cc:482] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.084985 14886 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.085094 14622 consensus_queue.cc:260] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.085397 14622 raft_consensus.cc:3037] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.087587 14886 raft_consensus.cc:504] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.088100 14622 raft_consensus.cc:2454] T d24bc6dd77904c7e9580939b27641500 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.088202 14886 leader_election.cc:288] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:11.089113 14882 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.089156 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d24bc6dd77904c7e9580939b27641500" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.089293 14882 raft_consensus.cc:482] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.089397 14882 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.092635 14882 raft_consensus.cc:504] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.093863 14882 leader_election.cc:288] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.093861 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.094079 14755 raft_consensus.cc:3032] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:16:11.094177 14755 raft_consensus.cc:729] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.094504 14755 consensus_queue.cc:260] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.094813 14755 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.094868 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.095016 14886 raft_consensus.cc:482] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.095131 14886 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.097672 14886 raft_consensus.cc:504] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:11.097929 14755 raft_consensus.cc:2396] T b6b42a4b52d94e1e9c72cdc2ab70aaca P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.099442 14886 leader_election.cc:288] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:11.100603 15013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.100824 15013 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.101053 14819 leader_election.cc:302] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters: 49e64758760942e1abdcdc0d2e1f07dd
I20240503 14:16:11.101902 14883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.102119 14883 raft_consensus.cc:482] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.102289 14883 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.104760 14883 raft_consensus.cc:504] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.105046 17201 raft_consensus.cc:2781] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:11.105545 17201 raft_consensus.cc:686] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.105810 17201 consensus_queue.cc:237] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.105875 14622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d761bca383e1488491576b8af4cc2c3f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.106415 14622 raft_consensus.cc:3037] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.105373 14819 leader_election.cc:302] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:11.107021 17053 raft_consensus.cc:2726] T 34c5b61b82b34beb8d10677e24e6f418 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 14:16:11.108232 14625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.108451 14625 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.108706 14483 catalog_manager.cc:5484] T d24bc6dd77904c7e9580939b27641500 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:11.109123 14622 raft_consensus.cc:2454] T d761bca383e1488491576b8af4cc2c3f P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 5.
I20240503 14:16:11.111359 14625 raft_consensus.cc:2396] T f145bdadec9d48dfb7370766e5196f4f P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.111806 14819 leader_election.cc:302] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:11.112103 17053 raft_consensus.cc:2781] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:11.112253 17053 raft_consensus.cc:686] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [term 5 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.112015 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b42a4b52d94e1e9c72cdc2ab70aaca" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.112496 17053 consensus_queue.cc:237] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:11.113025 15015 raft_consensus.cc:3037] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.114850 15012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.117157 14883 leader_election.cc:288] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.117446 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67818fe8ebae4b89bfa63f9222591c0a" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.119652 14886 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.119953 14886 raft_consensus.cc:482] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.120183 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f145bdadec9d48dfb7370766e5196f4f" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.120409 14755 raft_consensus.cc:3032] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:11.120523 14755 raft_consensus.cc:729] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.120277 14624 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.120587 14886 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.120993 14755 consensus_queue.cc:260] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.121459 14755 raft_consensus.cc:3037] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.124370 14886 raft_consensus.cc:504] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.116623 15012 raft_consensus.cc:3037] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:16:11.126920 14624 raft_consensus.cc:2454] T 67818fe8ebae4b89bfa63f9222591c0a P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 5.
I20240503 14:16:11.130970 15013 raft_consensus.cc:2396] T d761bca383e1488491576b8af4cc2c3f P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:11.127390 14819 leader_election.cc:302] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:11.134927 17201 raft_consensus.cc:2781] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:11.135120 17201 raft_consensus.cc:686] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [term 5 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.135442 17201 consensus_queue.cc:237] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.138522 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.138746 14624 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.141779 14753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc909a5952a944bfbc49a8052f73359c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.142024 14753 raft_consensus.cc:3032] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:11.142153 14753 raft_consensus.cc:729] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.142457 14753 consensus_queue.cc:260] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.142794 14753 raft_consensus.cc:3037] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.149075 14753 raft_consensus.cc:2396] T dc909a5952a944bfbc49a8052f73359c P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.155570 15012 raft_consensus.cc:2396] T 67818fe8ebae4b89bfa63f9222591c0a P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240503 14:16:11.156234 15015 raft_consensus.cc:2454] T b6b42a4b52d94e1e9c72cdc2ab70aaca P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 5.
I20240503 14:16:11.158341 16647 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.159157 14819 leader_election.cc:302] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:11.160144 17201 raft_consensus.cc:2781] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:16:11.160352 17201 raft_consensus.cc:686] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [term 5 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.160607 17201 consensus_queue.cc:237] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.163826 14755 raft_consensus.cc:2396] T f145bdadec9d48dfb7370766e5196f4f P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.164361 14820 leader_election.cc:302] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:11.164861 17201 raft_consensus.cc:2726] T f145bdadec9d48dfb7370766e5196f4f P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:11.168891 14886 leader_election.cc:288] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.170485 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.170799 14885 raft_consensus.cc:482] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.170974 14885 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.176455 14483 catalog_manager.cc:5484] T d761bca383e1488491576b8af4cc2c3f P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 3 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 5 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:11.179740 14885 raft_consensus.cc:504] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:11.181594 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.181892 15015 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.181993 14483 catalog_manager.cc:5484] T 67818fe8ebae4b89bfa63f9222591c0a P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 3 to 5, leader changed from 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 5 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:11.182466 14483 catalog_manager.cc:5484] T b6b42a4b52d94e1e9c72cdc2ab70aaca P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 4 to 5, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 5 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } }
I20240503 14:16:11.184810 15015 raft_consensus.cc:2454] T c97701fcd6674dadbb15881115dbc4f5 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.185966 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97701fcd6674dadbb15881115dbc4f5" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.186625 14755 raft_consensus.cc:3032] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:11.186761 14755 raft_consensus.cc:729] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.187000 14885 leader_election.cc:288] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.187350 14755 consensus_queue.cc:260] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:11.188299 14624 raft_consensus.cc:2396] T dc909a5952a944bfbc49a8052f73359c P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.193188 14755 raft_consensus.cc:3037] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.193637 14819 leader_election.cc:302] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:11.194038 17053 raft_consensus.cc:2781] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:11.194279 17053 raft_consensus.cc:686] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.194631 17053 consensus_queue.cc:237] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } }
I20240503 14:16:11.198701 14755 raft_consensus.cc:2396] T c97701fcd6674dadbb15881115dbc4f5 P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240503 14:16:11.198990 14996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.199642 14819 leader_election.cc:302] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:11.199962 17201 raft_consensus.cc:2726] T dc909a5952a944bfbc49a8052f73359c P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240503 14:16:11.205168 14735 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:16:11.220436 14483 catalog_manager.cc:5484] T c97701fcd6674dadbb15881115dbc4f5 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } }
I20240503 14:16:11.224218 16777 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.230407 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.230612 14885 raft_consensus.cc:482] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.230724 14885 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.233753 14885 raft_consensus.cc:504] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.234941 14885 leader_election.cc:288] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:11.235659 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.235884 14755 raft_consensus.cc:3032] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:16:11.235971 14755 raft_consensus.cc:729] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c04ff0b9952046aeb4aec65ee5f6b194, State: Running, Role: LEADER
I20240503 14:16:11.236294 14755 consensus_queue.cc:260] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.236591 14755 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.236692 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.236902 14885 raft_consensus.cc:482] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.237057 14885 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.238989 16515 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.239670 14755 raft_consensus.cc:2396] T 5fbeca22e62946dd922046cf5cdaef0c P c04ff0b9952046aeb4aec65ee5f6b194 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.239907 14885 raft_consensus.cc:504] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.240669 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca22e62946dd922046cf5cdaef0c" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.240868 14624 raft_consensus.cc:3037] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.241055 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.241250 15015 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.243798 14624 raft_consensus.cc:2396] T 5fbeca22e62946dd922046cf5cdaef0c P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b09f5cddfdf44e989f47a06c8da867f8 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.244341 15015 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.244513 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "054ce7121a9f4881aa2d38615f0c04a1" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.244684 14624 raft_consensus.cc:3032] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:11.244781 14624 raft_consensus.cc:729] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:11.244763 14819 leader_election.cc:302] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, b09f5cddfdf44e989f47a06c8da867f8; no voters:
I20240503 14:16:11.245082 14885 leader_election.cc:288] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:11.245055 14624 consensus_queue.cc:260] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.245401 14624 raft_consensus.cc:3037] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.245820 14820 leader_election.cc:302] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8; no voters: c04ff0b9952046aeb4aec65ee5f6b194, e9d4c39af431443189e8475561596810
I20240503 14:16:11.246078 17201 raft_consensus.cc:2781] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:11.246258 17201 raft_consensus.cc:686] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.246419 14885 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "843132c2c55645d598035edd0d2233c3"
dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
from {username='slave'} at 127.0.0.1:38290
I20240503 14:16:11.246589 14885 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.246727 14885 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.246750 17200 raft_consensus.cc:2726] T 5fbeca22e62946dd922046cf5cdaef0c P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 14:16:11.246505 17201 consensus_queue.cc:237] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.249799 14483 catalog_manager.cc:5484] T 054ce7121a9f4881aa2d38615f0c04a1 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:11.250149 14885 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.253067 14624 raft_consensus.cc:2454] T 054ce7121a9f4881aa2d38615f0c04a1 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.253885 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.254062 14624 raft_consensus.cc:3032] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Stepping down as leader of term 5
I20240503 14:16:11.254161 14624 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e9d4c39af431443189e8475561596810, State: Running, Role: LEADER
I20240503 14:16:11.254436 14624 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.254812 14624 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.259212 14624 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.259640 14819 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b09f5cddfdf44e989f47a06c8da867f8, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:11.259997 17053 raft_consensus.cc:2781] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Leader election won for term 6
I20240503 14:16:11.260149 17053 raft_consensus.cc:686] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Leader. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.260445 17053 consensus_queue.cc:237] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.262079 15015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "b09f5cddfdf44e989f47a06c8da867f8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
I20240503 14:16:11.262341 15015 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 5 FOLLOWER]: Advancing to term 6
I20240503 14:16:11.263397 14483 catalog_manager.cc:5484] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 reported cstate change: term changed from 5 to 6, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67). New cstate: current_term: 6 leader_uuid: "b09f5cddfdf44e989f47a06c8da867f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } }
I20240503 14:16:11.265847 15015 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b09f5cddfdf44e989f47a06c8da867f8 in term 6.
I20240503 14:16:11.266397 14885 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [CANDIDATE]: Term 6 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68:36453)
I20240503 14:16:11.275910 14865 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:16:11.278698 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:11.278895 15015 raft_consensus.cc:482] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.279011 15015 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.281728 15015 raft_consensus.cc:504] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.282969 15015 leader_election.cc:288] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 7 election: Requested vote from peers c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573), e9d4c39af431443189e8475561596810 (127.20.240.65:32989)
I20240503 14:16:11.283071 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.283314 14755 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.286043 14755 raft_consensus.cc:2454] T c8a328f25a3a4da4b4aa86b00ac015cd P c04ff0b9952046aeb4aec65ee5f6b194 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 7.
I20240503 14:16:11.286634 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:11.286746 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a328f25a3a4da4b4aa86b00ac015cd" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.286852 15015 raft_consensus.cc:482] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.286945 15015 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.286972 14624 raft_consensus.cc:3037] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.289758 15015 raft_consensus.cc:504] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.290344 14624 raft_consensus.cc:2396] T c8a328f25a3a4da4b4aa86b00ac015cd P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240503 14:16:11.290633 14950 leader_election.cc:302] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, c04ff0b9952046aeb4aec65ee5f6b194; no voters:
I20240503 14:16:11.290956 17098 raft_consensus.cc:2781] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Leader election won for term 7
I20240503 14:16:11.291101 17098 raft_consensus.cc:686] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [term 7 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:11.291211 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:11.291419 14885 raft_consensus.cc:3032] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:16:11.291486 14885 raft_consensus.cc:729] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.291412 17098 consensus_queue.cc:237] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } }
I20240503 14:16:11.291723 14885 consensus_queue.cc:260] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } }
I20240503 14:16:11.291982 14885 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.294323 14604 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.298544 16909 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:16:11.315017 14885 raft_consensus.cc:2396] T b9b1ac3afbb6471cb7172e438a1aa35e P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.317054 14755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9b1ac3afbb6471cb7172e438a1aa35e" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c04ff0b9952046aeb4aec65ee5f6b194"
I20240503 14:16:11.317289 14755 raft_consensus.cc:3037] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.320367 14755 raft_consensus.cc:2396] T b9b1ac3afbb6471cb7172e438a1aa35e P c04ff0b9952046aeb4aec65ee5f6b194 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240503 14:16:11.320900 14950 leader_election.cc:302] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd; no voters: b09f5cddfdf44e989f47a06c8da867f8, c04ff0b9952046aeb4aec65ee5f6b194
I20240503 14:16:11.321226 17097 raft_consensus.cc:2726] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240503 14:16:11.321296 15015 leader_election.cc:288] T b9b1ac3afbb6471cb7172e438a1aa35e P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 7 election: Requested vote from peers b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349), c04ff0b9952046aeb4aec65ee5f6b194 (127.20.240.66:36573)
I20240503 14:16:11.328042 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "843132c2c55645d598035edd0d2233c3"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:11.328244 15015 raft_consensus.cc:482] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.328374 15015 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.332389 15015 raft_consensus.cc:504] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.333529 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.333834 14624 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.336526 14885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "843132c2c55645d598035edd0d2233c3" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "b09f5cddfdf44e989f47a06c8da867f8"
I20240503 14:16:11.336735 14885 raft_consensus.cc:3032] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Stepping down as leader of term 6
I20240503 14:16:11.336808 14885 raft_consensus.cc:729] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b09f5cddfdf44e989f47a06c8da867f8, State: Running, Role: LEADER
I20240503 14:16:11.337003 14624 raft_consensus.cc:2454] T 843132c2c55645d598035edd0d2233c3 P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 7.
I20240503 14:16:11.337668 14949 leader_election.cc:302] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:11.337976 17097 raft_consensus.cc:2781] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Leader election won for term 7
I20240503 14:16:11.338260 14885 consensus_queue.cc:260] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.338543 14885 raft_consensus.cc:3037] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.338984 14483 catalog_manager.cc:5484] T c8a328f25a3a4da4b4aa86b00ac015cd P 49e64758760942e1abdcdc0d2e1f07dd reported cstate change: term changed from 5 to 7, leader changed from e9d4c39af431443189e8475561596810 (127.20.240.65) to 49e64758760942e1abdcdc0d2e1f07dd (127.20.240.68). New cstate: current_term: 7 leader_uuid: "49e64758760942e1abdcdc0d2e1f07dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c04ff0b9952046aeb4aec65ee5f6b194" member_type: VOTER last_known_addr { host: "127.20.240.66" port: 36573 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } } peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } }
I20240503 14:16:11.345211 14885 raft_consensus.cc:2396] T 843132c2c55645d598035edd0d2233c3 P b09f5cddfdf44e989f47a06c8da867f8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 49e64758760942e1abdcdc0d2e1f07dd for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240503 14:16:11.349373 15015 leader_election.cc:288] T 843132c2c55645d598035edd0d2233c3 P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 7 election: Requested vote from peers e9d4c39af431443189e8475561596810 (127.20.240.65:32989), b09f5cddfdf44e989f47a06c8da867f8 (127.20.240.67:44349)
I20240503 14:16:11.350517 15015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e"
dest_uuid: "49e64758760942e1abdcdc0d2e1f07dd"
from {username='slave'} at 127.0.0.1:41826
I20240503 14:16:11.350721 15015 raft_consensus.cc:482] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:16:11.350822 15015 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.353706 15015 raft_consensus.cc:504] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68" port: 36453 } }
I20240503 14:16:11.354843 14624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "435b95c19dfe45ceb5b15470a2ebd35e" candidate_uuid: "49e64758760942e1abdcdc0d2e1f07dd" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "e9d4c39af431443189e8475561596810"
I20240503 14:16:11.355067 14624 raft_consensus.cc:3037] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 6 FOLLOWER]: Advancing to term 7
I20240503 14:16:11.357996 14624 raft_consensus.cc:2454] T 435b95c19dfe45ceb5b15470a2ebd35e P e9d4c39af431443189e8475561596810 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e64758760942e1abdcdc0d2e1f07dd in term 7.
I20240503 14:16:11.358453 14949 leader_election.cc:302] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e64758760942e1abdcdc0d2e1f07dd, e9d4c39af431443189e8475561596810; no voters:
I20240503 14:16:11.358767 17098 raft_consensus.cc:2781] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 7 FOLLOWER]: Leader election won for term 7
I20240503 14:16:11.358889 17098 raft_consensus.cc:686] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [term 7 LEADER]: Becoming Leader. State: Replica: 49e64758760942e1abdcdc0d2e1f07dd, State: Running, Role: LEADER
I20240503 14:16:11.359174 17098 consensus_queue.cc:237] T 435b95c19dfe45ceb5b15470a2ebd35e P 49e64758760942e1abdcdc0d2e1f07dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9d4c39af431443189e8475561596810" member_type: VOTER last_known_addr { host: "127.20.240.65" port: 32989 } } peers { permanent_uuid: "b09f5cddfdf44e989f47a06c8da867f8" member_type: VOTER last_known_addr { host: "127.20.240.67" port: 44349 } } peers { permanent_uuid: "49e64758760942e1abdcdc0d2e1f07dd" member_type: VOTER last_known_addr { host: "127.20.240.68"